2023-03-27 04:55:51,062 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df 2023-03-27 04:55:51,072 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-27 04:55:51,084 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-27 04:55:51,085 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/cluster_6fb3ee64-1fe6-4ac3-e6f2-f169c384bd1a, deleteOnExit=true 2023-03-27 04:55:51,085 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-27 04:55:51,085 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/test.cache.data in system properties and HBase conf 2023-03-27 04:55:51,085 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/hadoop.tmp.dir in system properties and HBase conf 2023-03-27 04:55:51,086 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/hadoop.log.dir in system properties and HBase conf 2023-03-27 04:55:51,086 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-27 04:55:51,087 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-27 04:55:51,087 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-27 04:55:51,174 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-27 04:55:51,479 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-27 04:55:51,483 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-27 04:55:51,483 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-27 04:55:51,483 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-27 04:55:51,484 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-27 04:55:51,484 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-27 04:55:51,484 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-27 04:55:51,484 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-27 04:55:51,485 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-27 04:55:51,485 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-27 04:55:51,485 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/nfs.dump.dir in system properties and HBase conf 2023-03-27 04:55:51,486 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/java.io.tmpdir in system properties and HBase conf 2023-03-27 04:55:51,486 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-27 04:55:51,486 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-27 04:55:51,486 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-27 04:55:51,896 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-27 04:55:51,899 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-27 04:55:52,134 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-27 04:55:52,287 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-27 04:55:52,298 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-27 04:55:52,328 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-27 04:55:52,354 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/java.io.tmpdir/Jetty_localhost_localdomain_38587_hdfs____.vmhpdj/webapp 2023-03-27 04:55:52,456 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:38587 2023-03-27 04:55:52,489 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-27 04:55:52,490 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-27 04:55:52,800 WARN [Listener at localhost.localdomain/41275] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-27 04:55:52,887 WARN [Listener at localhost.localdomain/41275] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-27 04:55:52,901 WARN [Listener at localhost.localdomain/41275] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-27 04:55:52,906 INFO [Listener at localhost.localdomain/41275] log.Slf4jLog(67): jetty-6.1.26 2023-03-27 04:55:52,910 INFO [Listener at localhost.localdomain/41275] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/java.io.tmpdir/Jetty_localhost_46507_datanode____.huue3j/webapp 2023-03-27 04:55:52,985 INFO [Listener at localhost.localdomain/41275] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46507 2023-03-27 04:55:53,250 WARN [Listener at localhost.localdomain/43311] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-27 04:55:53,574 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc043c01cac85f0bc: Processing first storage report for DS-0e35c55b-c2e0-434e-bb54-7998f572cc6e from datanode 093b1e1f-e2c0-4364-97ab-7016262b240a 2023-03-27 04:55:53,576 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc043c01cac85f0bc: from storage DS-0e35c55b-c2e0-434e-bb54-7998f572cc6e node DatanodeRegistration(127.0.0.1:46667, datanodeUuid=093b1e1f-e2c0-4364-97ab-7016262b240a, infoPort=43159, infoSecurePort=0, ipcPort=43311, storageInfo=lv=-57;cid=testClusterID;nsid=1462856770;c=1679892951960), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-03-27 04:55:53,576 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc043c01cac85f0bc: Processing first storage report for DS-f6950077-e404-4ad1-bee0-92f2e9715e83 from datanode 093b1e1f-e2c0-4364-97ab-7016262b240a 2023-03-27 04:55:53,576 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc043c01cac85f0bc: from storage DS-f6950077-e404-4ad1-bee0-92f2e9715e83 node DatanodeRegistration(127.0.0.1:46667, datanodeUuid=093b1e1f-e2c0-4364-97ab-7016262b240a, infoPort=43159, infoSecurePort=0, ipcPort=43311, storageInfo=lv=-57;cid=testClusterID;nsid=1462856770;c=1679892951960), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-03-27 04:55:53,772 DEBUG [Listener at localhost.localdomain/43311] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df 2023-03-27 04:55:53,821 INFO [Listener at localhost.localdomain/43311] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/cluster_6fb3ee64-1fe6-4ac3-e6f2-f169c384bd1a/zookeeper_0, clientPort=61059, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/cluster_6fb3ee64-1fe6-4ac3-e6f2-f169c384bd1a/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/cluster_6fb3ee64-1fe6-4ac3-e6f2-f169c384bd1a/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-27 04:55:53,832 INFO [Listener at localhost.localdomain/43311] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=61059 2023-03-27 04:55:53,842 INFO [Listener at localhost.localdomain/43311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-27 04:55:53,845 INFO [Listener at localhost.localdomain/43311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-27 04:55:54,453 INFO [Listener at localhost.localdomain/43311] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c with version=8 2023-03-27 04:55:54,454 INFO [Listener at localhost.localdomain/43311] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/hbase-staging 2023-03-27 04:55:54,694 INFO [Listener at localhost.localdomain/43311] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-27 04:55:55,039 INFO [Listener at localhost.localdomain/43311] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-03-27 04:55:55,063 INFO [Listener at localhost.localdomain/43311] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-27 04:55:55,064 INFO [Listener at localhost.localdomain/43311] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-27 04:55:55,064 INFO [Listener at localhost.localdomain/43311] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-27 04:55:55,064 INFO [Listener at localhost.localdomain/43311] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-27 04:55:55,064 INFO [Listener at localhost.localdomain/43311] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-27 04:55:55,177 INFO [Listener at localhost.localdomain/43311] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-27 04:55:55,236 DEBUG [Listener at localhost.localdomain/43311] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-27 04:55:55,322 INFO [Listener at localhost.localdomain/43311] ipc.NettyRpcServer(120): Bind to /148.251.75.209:36293 2023-03-27 04:55:55,331 INFO [Listener at localhost.localdomain/43311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-27 04:55:55,333 INFO [Listener at localhost.localdomain/43311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-27 04:55:55,351 INFO [Listener at localhost.localdomain/43311] zookeeper.RecoverableZooKeeper(93): Process identifier=master:36293 connecting to ZooKeeper ensemble=127.0.0.1:61059 2023-03-27 04:55:55,384 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:362930x0, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-27 04:55:55,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:36293-0x100512252fd0000 connected 2023-03-27 04:55:55,432 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-27 04:55:55,433 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-27 04:55:55,437 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-27 04:55:55,443 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36293 2023-03-27 04:55:55,444 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36293 2023-03-27 04:55:55,444 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36293 2023-03-27 04:55:55,446 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36293 2023-03-27 04:55:55,446 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36293 2023-03-27 04:55:55,451 INFO [Listener at localhost.localdomain/43311] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c, hbase.cluster.distributed=false 2023-03-27 04:55:55,506 INFO [Listener at localhost.localdomain/43311] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-03-27 04:55:55,506 INFO [Listener at localhost.localdomain/43311] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-27 04:55:55,506 INFO [Listener at localhost.localdomain/43311] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-27 04:55:55,507 INFO [Listener at localhost.localdomain/43311] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-27 04:55:55,507 INFO [Listener at localhost.localdomain/43311] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-27 04:55:55,507 INFO [Listener at localhost.localdomain/43311] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-27 04:55:55,511 INFO [Listener at localhost.localdomain/43311] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-27 04:55:55,513 INFO [Listener at localhost.localdomain/43311] ipc.NettyRpcServer(120): Bind to /148.251.75.209:44923 2023-03-27 04:55:55,515 INFO [Listener at localhost.localdomain/43311] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-27 04:55:55,520 DEBUG [Listener at localhost.localdomain/43311] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-27 04:55:55,520 INFO [Listener at localhost.localdomain/43311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-27 04:55:55,523 INFO [Listener at localhost.localdomain/43311] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-27 04:55:55,524 INFO [Listener at localhost.localdomain/43311] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:44923 connecting to ZooKeeper ensemble=127.0.0.1:61059 2023-03-27 04:55:55,528 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:449230x0, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-27 04:55:55,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:44923-0x100512252fd0001 connected 2023-03-27 04:55:55,529 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-27 04:55:55,530 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-27 04:55:55,531 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-27 04:55:55,532 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44923 2023-03-27 04:55:55,532 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44923 2023-03-27 04:55:55,532 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44923 2023-03-27 04:55:55,533 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44923 2023-03-27 04:55:55,533 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44923 2023-03-27 04:55:55,534 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,36293,1679892954567 2023-03-27 04:55:55,543 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-27 04:55:55,544 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,36293,1679892954567 2023-03-27 04:55:55,560 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-27 04:55:55,560 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-27 04:55:55,560 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-27 04:55:55,561 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-27 04:55:55,562 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,36293,1679892954567 from backup master directory 2023-03-27 04:55:55,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-27 04:55:55,565 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,36293,1679892954567 2023-03-27 04:55:55,565 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-27 04:55:55,566 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-27 04:55:55,566 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,36293,1679892954567 2023-03-27 04:55:55,568 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-27 04:55:55,569 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-27 04:55:55,643 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/hbase.id with ID: daa3d689-f31e-48f7-aec7-8e3e2cd4151c 2023-03-27 04:55:55,687 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-27 04:55:55,700 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-27 04:55:55,737 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7bb49698 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:55:55,767 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8224b85, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:55:55,787 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-27 04:55:55,788 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-27 04:55:55,803 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-27 04:55:55,803 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-27 04:55:55,805 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:55:55,808 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:55:55,809 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-27 04:55:55,837 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/MasterData/data/master/store-tmp 2023-03-27 04:55:56,265 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:55:56,266 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-27 04:55:56,266 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-27 04:55:56,266 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-27 04:55:56,266 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-27 04:55:56,266 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-27 04:55:56,266 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-27 04:55:56,267 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-27 04:55:56,269 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/MasterData/WALs/jenkins-hbase20.apache.org,36293,1679892954567 2023-03-27 04:55:56,298 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C36293%2C1679892954567, suffix=, logDir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/MasterData/WALs/jenkins-hbase20.apache.org,36293,1679892954567, archiveDir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/MasterData/oldWALs, maxLogs=10 2023-03-27 04:55:56,348 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46667,DS-0e35c55b-c2e0-434e-bb54-7998f572cc6e,DISK] 2023-03-27 04:55:56,355 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:55:56,407 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/MasterData/WALs/jenkins-hbase20.apache.org,36293,1679892954567/jenkins-hbase20.apache.org%2C36293%2C1679892954567.1679892956308 2023-03-27 04:55:56,408 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46667,DS-0e35c55b-c2e0-434e-bb54-7998f572cc6e,DISK]] 2023-03-27 04:55:56,408 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-27 04:55:56,408 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:55:56,411 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-27 04:55:56,412 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-27 04:55:56,465 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-27 04:55:56,471 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-27 04:55:56,492 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-27 04:55:56,505 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:55:56,511 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-27 04:55:56,513 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-27 04:55:56,532 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-27 04:55:56,540 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-27 04:55:56,542 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61279417, jitterRate=-0.08686552941799164}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-27 04:55:56,542 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-27 04:55:56,544 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-27 04:55:56,563 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-27 04:55:56,563 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-27 04:55:56,565 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-27 04:55:56,567 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-27 04:55:56,594 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 27 msec 2023-03-27 04:55:56,594 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-27 04:55:56,623 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-27 04:55:56,629 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-27 04:55:56,652 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-27 04:55:56,655 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-27 04:55:56,657 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-27 04:55:56,662 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-27 04:55:56,665 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-27 04:55:56,668 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-27 04:55:56,669 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-27 04:55:56,670 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-27 04:55:56,686 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-27 04:55:56,691 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-27 04:55:56,691 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-27 04:55:56,691 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-27 04:55:56,691 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,36293,1679892954567, sessionid=0x100512252fd0000, setting cluster-up flag (Was=false) 2023-03-27 04:55:56,704 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-27 04:55:56,708 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-27 04:55:56,709 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,36293,1679892954567 2023-03-27 04:55:56,723 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-27 04:55:56,727 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-27 04:55:56,729 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,36293,1679892954567 2023-03-27 04:55:56,731 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.hbase-snapshot/.tmp 2023-03-27 04:55:56,737 INFO [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(952): ClusterId : daa3d689-f31e-48f7-aec7-8e3e2cd4151c 2023-03-27 04:55:56,741 DEBUG [RS:0;jenkins-hbase20:44923] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-27 04:55:56,746 DEBUG [RS:0;jenkins-hbase20:44923] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-27 04:55:56,746 DEBUG [RS:0;jenkins-hbase20:44923] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-27 04:55:56,748 DEBUG [RS:0;jenkins-hbase20:44923] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-27 04:55:56,749 DEBUG [RS:0;jenkins-hbase20:44923] zookeeper.ReadOnlyZKClient(139): Connect 0x140e3e31 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:55:56,754 DEBUG [RS:0;jenkins-hbase20:44923] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ac9c2c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:55:56,755 DEBUG [RS:0;jenkins-hbase20:44923] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ef65541, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-03-27 04:55:56,787 DEBUG [RS:0;jenkins-hbase20:44923] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:44923 2023-03-27 04:55:56,792 INFO [RS:0;jenkins-hbase20:44923] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-27 04:55:56,792 INFO [RS:0;jenkins-hbase20:44923] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-27 04:55:56,792 DEBUG [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(1023): About to register with Master. 2023-03-27 04:55:56,795 INFO [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,36293,1679892954567 with isa=jenkins-hbase20.apache.org/148.251.75.209:44923, startcode=1679892955505 2023-03-27 04:55:56,815 DEBUG [RS:0;jenkins-hbase20:44923] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-27 04:55:56,868 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-27 04:55:56,877 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54055, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-27 04:55:56,879 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-27 04:55:56,879 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-27 04:55:56,879 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-27 04:55:56,879 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-27 04:55:56,879 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-03-27 04:55:56,879 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-27 04:55:56,879 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-03-27 04:55:56,880 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-27 04:55:56,881 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1679892986881 2023-03-27 04:55:56,884 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-27 04:55:56,886 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-27 04:55:56,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:55:56,887 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-27 04:55:56,891 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-27 04:55:56,897 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-27 04:55:56,906 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-27 04:55:56,906 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-27 04:55:56,907 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-27 04:55:56,907 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-27 04:55:56,908 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-27 04:55:56,909 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-27 04:55:56,911 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-27 04:55:56,911 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-27 04:55:56,914 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-27 04:55:56,914 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-27 04:55:56,918 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1679892956917,5,FailOnTimeoutGroup] 2023-03-27 04:55:56,919 DEBUG [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(2829): Master is not running yet 2023-03-27 04:55:56,919 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1679892956918,5,FailOnTimeoutGroup] 2023-03-27 04:55:56,920 WARN [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-03-27 04:55:56,920 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-27 04:55:56,921 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-27 04:55:56,926 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-27 04:55:56,927 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-27 04:55:56,930 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-27 04:55:56,931 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-27 04:55:56,931 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c 2023-03-27 04:55:56,952 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:55:56,955 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-27 04:55:56,958 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/info 2023-03-27 04:55:56,958 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-27 04:55:56,960 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:55:56,960 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-27 04:55:56,966 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/rep_barrier 2023-03-27 04:55:56,967 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-27 04:55:56,968 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:55:56,969 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-27 04:55:56,972 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/table 2023-03-27 04:55:56,973 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-27 04:55:56,974 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:55:56,976 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740 2023-03-27 04:55:56,978 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740 2023-03-27 04:55:56,982 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-27 04:55:56,985 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-27 04:55:56,989 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-27 04:55:56,991 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74617777, jitterRate=0.11189152300357819}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-27 04:55:56,991 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-27 04:55:56,991 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-27 04:55:56,991 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-27 04:55:56,991 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-27 04:55:56,991 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-27 04:55:56,991 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-27 04:55:56,992 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-27 04:55:56,993 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-27 04:55:56,997 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-27 04:55:56,998 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-27 04:55:57,008 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-27 04:55:57,022 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-27 04:55:57,023 INFO [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,36293,1679892954567 with isa=jenkins-hbase20.apache.org/148.251.75.209:44923, startcode=1679892955505 2023-03-27 04:55:57,025 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-27 04:55:57,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:57,034 DEBUG [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c 2023-03-27 04:55:57,034 DEBUG [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:41275 2023-03-27 04:55:57,035 DEBUG [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-27 04:55:57,040 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-27 04:55:57,041 DEBUG [RS:0;jenkins-hbase20:44923] zookeeper.ZKUtil(162): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:57,042 WARN [RS:0;jenkins-hbase20:44923] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-27 04:55:57,042 INFO [RS:0;jenkins-hbase20:44923] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-27 04:55:57,042 DEBUG [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/WALs/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:57,043 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,44923,1679892955505] 2023-03-27 04:55:57,050 DEBUG [RS:0;jenkins-hbase20:44923] zookeeper.ZKUtil(162): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:57,059 DEBUG [RS:0;jenkins-hbase20:44923] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-27 04:55:57,066 INFO [RS:0;jenkins-hbase20:44923] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-27 04:55:57,091 INFO [RS:0;jenkins-hbase20:44923] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-27 04:55:57,094 INFO [RS:0;jenkins-hbase20:44923] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-27 04:55:57,095 INFO [RS:0;jenkins-hbase20:44923] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-27 04:55:57,096 INFO [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-27 04:55:57,102 INFO [RS:0;jenkins-hbase20:44923] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-27 04:55:57,103 DEBUG [RS:0;jenkins-hbase20:44923] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-27 04:55:57,103 DEBUG [RS:0;jenkins-hbase20:44923] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-27 04:55:57,104 DEBUG [RS:0;jenkins-hbase20:44923] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-27 04:55:57,104 DEBUG [RS:0;jenkins-hbase20:44923] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-27 04:55:57,104 DEBUG [RS:0;jenkins-hbase20:44923] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-27 04:55:57,104 DEBUG [RS:0;jenkins-hbase20:44923] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-03-27 04:55:57,104 DEBUG [RS:0;jenkins-hbase20:44923] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-27 04:55:57,105 DEBUG [RS:0;jenkins-hbase20:44923] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-27 04:55:57,105 DEBUG [RS:0;jenkins-hbase20:44923] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-27 04:55:57,105 DEBUG [RS:0;jenkins-hbase20:44923] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-27 04:55:57,106 INFO [RS:0;jenkins-hbase20:44923] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-27 04:55:57,106 INFO [RS:0;jenkins-hbase20:44923] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-27 04:55:57,106 INFO [RS:0;jenkins-hbase20:44923] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-27 04:55:57,122 INFO [RS:0;jenkins-hbase20:44923] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-27 04:55:57,124 INFO [RS:0;jenkins-hbase20:44923] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44923,1679892955505-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-27 04:55:57,139 INFO [RS:0;jenkins-hbase20:44923] regionserver.Replication(203): jenkins-hbase20.apache.org,44923,1679892955505 started 2023-03-27 04:55:57,139 INFO [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,44923,1679892955505, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:44923, sessionid=0x100512252fd0001 2023-03-27 04:55:57,140 DEBUG [RS:0;jenkins-hbase20:44923] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-27 04:55:57,140 DEBUG [RS:0;jenkins-hbase20:44923] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:57,140 DEBUG [RS:0;jenkins-hbase20:44923] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,44923,1679892955505' 2023-03-27 04:55:57,140 DEBUG [RS:0;jenkins-hbase20:44923] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:55:57,141 DEBUG [RS:0;jenkins-hbase20:44923] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:55:57,141 DEBUG [RS:0;jenkins-hbase20:44923] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-27 04:55:57,141 DEBUG [RS:0;jenkins-hbase20:44923] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-27 04:55:57,141 DEBUG [RS:0;jenkins-hbase20:44923] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:57,141 DEBUG [RS:0;jenkins-hbase20:44923] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,44923,1679892955505' 2023-03-27 04:55:57,141 DEBUG [RS:0;jenkins-hbase20:44923] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-27 04:55:57,142 DEBUG [RS:0;jenkins-hbase20:44923] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-27 04:55:57,142 DEBUG [RS:0;jenkins-hbase20:44923] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-27 04:55:57,142 INFO [RS:0;jenkins-hbase20:44923] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-27 04:55:57,142 INFO [RS:0;jenkins-hbase20:44923] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-27 04:55:57,178 DEBUG [jenkins-hbase20:36293] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-27 04:55:57,180 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,44923,1679892955505, state=OPENING 2023-03-27 04:55:57,186 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-27 04:55:57,187 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-27 04:55:57,188 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-27 04:55:57,191 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:55:57,252 INFO [RS:0;jenkins-hbase20:44923] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C44923%2C1679892955505, suffix=, logDir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/WALs/jenkins-hbase20.apache.org,44923,1679892955505, archiveDir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/oldWALs, maxLogs=32 2023-03-27 04:55:57,271 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46667,DS-0e35c55b-c2e0-434e-bb54-7998f572cc6e,DISK] 2023-03-27 04:55:57,275 INFO [RS:0;jenkins-hbase20:44923] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/WALs/jenkins-hbase20.apache.org,44923,1679892955505/jenkins-hbase20.apache.org%2C44923%2C1679892955505.1679892957255 2023-03-27 04:55:57,275 DEBUG [RS:0;jenkins-hbase20:44923] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46667,DS-0e35c55b-c2e0-434e-bb54-7998f572cc6e,DISK]] 2023-03-27 04:55:57,373 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:57,375 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-27 04:55:57,379 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49444, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-27 04:55:57,390 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-27 04:55:57,390 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-27 04:55:57,394 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C44923%2C1679892955505.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/WALs/jenkins-hbase20.apache.org,44923,1679892955505, archiveDir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/oldWALs, maxLogs=32 2023-03-27 04:55:57,409 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46667,DS-0e35c55b-c2e0-434e-bb54-7998f572cc6e,DISK] 2023-03-27 04:55:57,413 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/WALs/jenkins-hbase20.apache.org,44923,1679892955505/jenkins-hbase20.apache.org%2C44923%2C1679892955505.meta.1679892957395.meta 2023-03-27 04:55:57,414 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46667,DS-0e35c55b-c2e0-434e-bb54-7998f572cc6e,DISK]] 2023-03-27 04:55:57,414 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-27 04:55:57,416 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-27 04:55:57,431 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-27 04:55:57,436 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-27 04:55:57,440 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-27 04:55:57,440 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:55:57,440 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-27 04:55:57,441 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-27 04:55:57,443 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-27 04:55:57,445 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/info 2023-03-27 04:55:57,445 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/info 2023-03-27 04:55:57,446 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-27 04:55:57,447 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:55:57,447 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-27 04:55:57,448 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/rep_barrier 2023-03-27 04:55:57,449 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/rep_barrier 2023-03-27 04:55:57,449 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-27 04:55:57,450 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:55:57,450 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-27 04:55:57,452 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/table 2023-03-27 04:55:57,452 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/table 2023-03-27 04:55:57,452 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-27 04:55:57,453 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:55:57,455 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740 2023-03-27 04:55:57,457 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740 2023-03-27 04:55:57,460 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-27 04:55:57,463 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-27 04:55:57,464 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69556057, jitterRate=0.03646601736545563}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-27 04:55:57,464 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-27 04:55:57,472 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1679892957365 2023-03-27 04:55:57,489 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-27 04:55:57,490 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-27 04:55:57,491 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,44923,1679892955505, state=OPEN 2023-03-27 04:55:57,494 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-27 04:55:57,494 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-27 04:55:57,499 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-27 04:55:57,499 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,44923,1679892955505 in 303 msec 2023-03-27 04:55:57,510 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-27 04:55:57,510 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 493 msec 2023-03-27 04:55:57,516 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 726 msec 2023-03-27 04:55:57,517 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1679892957517, completionTime=-1 2023-03-27 04:55:57,517 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-27 04:55:57,517 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-03-27 04:55:57,574 DEBUG [hconnection-0x5749f0e9-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:55:57,577 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49448, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:55:57,597 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-03-27 04:55:57,597 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1679893017597 2023-03-27 04:55:57,597 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1679893077597 2023-03-27 04:55:57,597 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 80 msec 2023-03-27 04:55:57,634 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,36293,1679892954567-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-27 04:55:57,634 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,36293,1679892954567-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-27 04:55:57,634 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,36293,1679892954567-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-27 04:55:57,636 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:36293, period=300000, unit=MILLISECONDS is enabled. 2023-03-27 04:55:57,636 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-27 04:55:57,643 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-27 04:55:57,650 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-27 04:55:57,651 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-27 04:55:57,661 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-27 04:55:57,664 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-27 04:55:57,667 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-27 04:55:57,688 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/hbase/namespace/44fb63855dd55c4d7b8c984f4960cd12 2023-03-27 04:55:57,690 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/hbase/namespace/44fb63855dd55c4d7b8c984f4960cd12 empty. 2023-03-27 04:55:57,691 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/hbase/namespace/44fb63855dd55c4d7b8c984f4960cd12 2023-03-27 04:55:57,691 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-27 04:55:58,119 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-27 04:55:58,121 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 44fb63855dd55c4d7b8c984f4960cd12, NAME => 'hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp 2023-03-27 04:55:58,134 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:55:58,134 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 44fb63855dd55c4d7b8c984f4960cd12, disabling compactions & flushes 2023-03-27 04:55:58,134 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12. 2023-03-27 04:55:58,135 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12. 2023-03-27 04:55:58,135 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12. after waiting 0 ms 2023-03-27 04:55:58,135 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12. 2023-03-27 04:55:58,135 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12. 2023-03-27 04:55:58,135 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 44fb63855dd55c4d7b8c984f4960cd12: 2023-03-27 04:55:58,139 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-27 04:55:58,153 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679892958142"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679892958142"}]},"ts":"1679892958142"} 2023-03-27 04:55:58,176 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-27 04:55:58,179 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-27 04:55:58,183 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679892958179"}]},"ts":"1679892958179"} 2023-03-27 04:55:58,188 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-27 04:55:58,195 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=44fb63855dd55c4d7b8c984f4960cd12, ASSIGN}] 2023-03-27 04:55:58,198 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=44fb63855dd55c4d7b8c984f4960cd12, ASSIGN 2023-03-27 04:55:58,200 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=44fb63855dd55c4d7b8c984f4960cd12, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,44923,1679892955505; forceNewPlan=false, retain=false 2023-03-27 04:55:58,351 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=44fb63855dd55c4d7b8c984f4960cd12, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:58,352 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679892958351"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679892958351"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679892958351"}]},"ts":"1679892958351"} 2023-03-27 04:55:58,360 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 44fb63855dd55c4d7b8c984f4960cd12, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:55:58,520 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12. 2023-03-27 04:55:58,521 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 44fb63855dd55c4d7b8c984f4960cd12, NAME => 'hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12.', STARTKEY => '', ENDKEY => ''} 2023-03-27 04:55:58,522 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 44fb63855dd55c4d7b8c984f4960cd12 2023-03-27 04:55:58,522 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:55:58,522 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 44fb63855dd55c4d7b8c984f4960cd12 2023-03-27 04:55:58,522 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 44fb63855dd55c4d7b8c984f4960cd12 2023-03-27 04:55:58,524 INFO [StoreOpener-44fb63855dd55c4d7b8c984f4960cd12-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 44fb63855dd55c4d7b8c984f4960cd12 2023-03-27 04:55:58,527 DEBUG [StoreOpener-44fb63855dd55c4d7b8c984f4960cd12-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/namespace/44fb63855dd55c4d7b8c984f4960cd12/info 2023-03-27 04:55:58,527 DEBUG [StoreOpener-44fb63855dd55c4d7b8c984f4960cd12-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/namespace/44fb63855dd55c4d7b8c984f4960cd12/info 2023-03-27 04:55:58,528 INFO [StoreOpener-44fb63855dd55c4d7b8c984f4960cd12-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 44fb63855dd55c4d7b8c984f4960cd12 columnFamilyName info 2023-03-27 04:55:58,528 INFO [StoreOpener-44fb63855dd55c4d7b8c984f4960cd12-1] regionserver.HStore(310): Store=44fb63855dd55c4d7b8c984f4960cd12/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:55:58,530 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/namespace/44fb63855dd55c4d7b8c984f4960cd12 2023-03-27 04:55:58,531 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/namespace/44fb63855dd55c4d7b8c984f4960cd12 2023-03-27 04:55:58,536 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 44fb63855dd55c4d7b8c984f4960cd12 2023-03-27 04:55:58,540 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/namespace/44fb63855dd55c4d7b8c984f4960cd12/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-27 04:55:58,541 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 44fb63855dd55c4d7b8c984f4960cd12; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72922263, jitterRate=0.08662639558315277}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-27 04:55:58,541 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 44fb63855dd55c4d7b8c984f4960cd12: 2023-03-27 04:55:58,544 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12., pid=6, masterSystemTime=1679892958514 2023-03-27 04:55:58,548 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12. 2023-03-27 04:55:58,548 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12. 2023-03-27 04:55:58,550 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=44fb63855dd55c4d7b8c984f4960cd12, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:58,551 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679892958550"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679892958550"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679892958550"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679892958550"}]},"ts":"1679892958550"} 2023-03-27 04:55:58,560 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-27 04:55:58,560 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 44fb63855dd55c4d7b8c984f4960cd12, server=jenkins-hbase20.apache.org,44923,1679892955505 in 196 msec 2023-03-27 04:55:58,564 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-27 04:55:58,565 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=44fb63855dd55c4d7b8c984f4960cd12, ASSIGN in 366 msec 2023-03-27 04:55:58,566 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-27 04:55:58,567 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679892958567"}]},"ts":"1679892958567"} 2023-03-27 04:55:58,571 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-27 04:55:58,574 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-27 04:55:58,577 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 921 msec 2023-03-27 04:55:58,665 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-27 04:55:58,667 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-27 04:55:58,667 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-27 04:55:58,709 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-27 04:55:58,731 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-27 04:55:58,745 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 43 msec 2023-03-27 04:55:58,757 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-27 04:55:58,772 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-27 04:55:58,777 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 20 msec 2023-03-27 04:55:58,797 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-27 04:55:58,798 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-27 04:55:58,798 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.232sec 2023-03-27 04:55:58,801 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-27 04:55:58,802 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-27 04:55:58,802 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-27 04:55:58,804 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,36293,1679892954567-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-27 04:55:58,804 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,36293,1679892954567-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-27 04:55:58,824 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-27 04:55:58,845 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x70531455 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:55:58,850 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a972b34, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:55:58,863 DEBUG [hconnection-0x5e53e335-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:55:58,882 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49456, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:55:58,898 INFO [Listener at localhost.localdomain/43311] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,36293,1679892954567 2023-03-27 04:55:58,919 INFO [Listener at localhost.localdomain/43311] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=263, ProcessCount=167, AvailableMemoryMB=2024 2023-03-27 04:55:58,940 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-27 04:55:58,946 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53176, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-27 04:55:58,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-27 04:55:58,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-27 04:55:58,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-27 04:55:58,966 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-27 04:55:58,969 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-27 04:55:58,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-27 04:55:58,974 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79 2023-03-27 04:55:58,975 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79 empty. 2023-03-27 04:55:58,977 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79 2023-03-27 04:55:58,977 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-27 04:55:58,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-27 04:55:59,002 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-27 04:55:59,004 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e7756c618a32a862ad613327bee74d79, NAME => 'TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp 2023-03-27 04:55:59,019 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:55:59,019 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e7756c618a32a862ad613327bee74d79, disabling compactions & flushes 2023-03-27 04:55:59,019 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:55:59,019 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:55:59,019 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. after waiting 0 ms 2023-03-27 04:55:59,019 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:55:59,019 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:55:59,019 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:55:59,024 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-27 04:55:59,026 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679892959026"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679892959026"}]},"ts":"1679892959026"} 2023-03-27 04:55:59,029 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-27 04:55:59,031 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-27 04:55:59,032 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679892959032"}]},"ts":"1679892959032"} 2023-03-27 04:55:59,035 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-27 04:55:59,039 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e7756c618a32a862ad613327bee74d79, ASSIGN}] 2023-03-27 04:55:59,041 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e7756c618a32a862ad613327bee74d79, ASSIGN 2023-03-27 04:55:59,043 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e7756c618a32a862ad613327bee74d79, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,44923,1679892955505; forceNewPlan=false, retain=false 2023-03-27 04:55:59,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-27 04:55:59,194 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=e7756c618a32a862ad613327bee74d79, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,195 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679892959194"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679892959194"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679892959194"}]},"ts":"1679892959194"} 2023-03-27 04:55:59,200 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:55:59,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-27 04:55:59,361 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:55:59,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e7756c618a32a862ad613327bee74d79, NAME => 'TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.', STARTKEY => '', ENDKEY => ''} 2023-03-27 04:55:59,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e7756c618a32a862ad613327bee74d79 2023-03-27 04:55:59,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:55:59,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for e7756c618a32a862ad613327bee74d79 2023-03-27 04:55:59,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for e7756c618a32a862ad613327bee74d79 2023-03-27 04:55:59,364 INFO [StoreOpener-e7756c618a32a862ad613327bee74d79-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e7756c618a32a862ad613327bee74d79 2023-03-27 04:55:59,366 DEBUG [StoreOpener-e7756c618a32a862ad613327bee74d79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A 2023-03-27 04:55:59,366 DEBUG [StoreOpener-e7756c618a32a862ad613327bee74d79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A 2023-03-27 04:55:59,367 INFO [StoreOpener-e7756c618a32a862ad613327bee74d79-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:55:59,368 INFO [StoreOpener-e7756c618a32a862ad613327bee74d79-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7756c618a32a862ad613327bee74d79 columnFamilyName A 2023-03-27 04:55:59,369 INFO [StoreOpener-e7756c618a32a862ad613327bee74d79-1] regionserver.HStore(310): Store=e7756c618a32a862ad613327bee74d79/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:55:59,369 INFO [StoreOpener-e7756c618a32a862ad613327bee74d79-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e7756c618a32a862ad613327bee74d79 2023-03-27 04:55:59,371 DEBUG [StoreOpener-e7756c618a32a862ad613327bee74d79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B 2023-03-27 04:55:59,371 DEBUG [StoreOpener-e7756c618a32a862ad613327bee74d79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B 2023-03-27 04:55:59,372 INFO [StoreOpener-e7756c618a32a862ad613327bee74d79-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:55:59,372 INFO [StoreOpener-e7756c618a32a862ad613327bee74d79-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7756c618a32a862ad613327bee74d79 columnFamilyName B 2023-03-27 04:55:59,374 INFO [StoreOpener-e7756c618a32a862ad613327bee74d79-1] regionserver.HStore(310): Store=e7756c618a32a862ad613327bee74d79/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:55:59,374 INFO [StoreOpener-e7756c618a32a862ad613327bee74d79-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e7756c618a32a862ad613327bee74d79 2023-03-27 04:55:59,376 DEBUG [StoreOpener-e7756c618a32a862ad613327bee74d79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C 2023-03-27 04:55:59,376 DEBUG [StoreOpener-e7756c618a32a862ad613327bee74d79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C 2023-03-27 04:55:59,376 INFO [StoreOpener-e7756c618a32a862ad613327bee74d79-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:55:59,377 INFO [StoreOpener-e7756c618a32a862ad613327bee74d79-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e7756c618a32a862ad613327bee74d79 columnFamilyName C 2023-03-27 04:55:59,378 INFO [StoreOpener-e7756c618a32a862ad613327bee74d79-1] regionserver.HStore(310): Store=e7756c618a32a862ad613327bee74d79/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:55:59,378 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:55:59,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79 2023-03-27 04:55:59,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79 2023-03-27 04:55:59,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-27 04:55:59,387 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for e7756c618a32a862ad613327bee74d79 2023-03-27 04:55:59,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-27 04:55:59,392 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened e7756c618a32a862ad613327bee74d79; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68109853, jitterRate=0.014915898442268372}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-27 04:55:59,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:55:59,393 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., pid=11, masterSystemTime=1679892959354 2023-03-27 04:55:59,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:55:59,395 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:55:59,396 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=e7756c618a32a862ad613327bee74d79, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,396 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679892959396"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679892959396"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679892959396"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679892959396"}]},"ts":"1679892959396"} 2023-03-27 04:55:59,402 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-27 04:55:59,402 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 in 199 msec 2023-03-27 04:55:59,406 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-27 04:55:59,407 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e7756c618a32a862ad613327bee74d79, ASSIGN in 363 msec 2023-03-27 04:55:59,408 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-27 04:55:59,408 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679892959408"}]},"ts":"1679892959408"} 2023-03-27 04:55:59,410 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-27 04:55:59,414 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-27 04:55:59,416 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 453 msec 2023-03-27 04:55:59,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-27 04:55:59,594 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-27 04:55:59,602 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x12582a26 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:55:59,607 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@458f9a9e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:55:59,609 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:55:59,612 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49472, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:55:59,615 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-27 04:55:59,618 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53180, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-27 04:55:59,626 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x3e0ba002 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:55:59,630 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20e4e4f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:55:59,630 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x77443fb0 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:55:59,635 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13e837e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:55:59,635 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x136956fd to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:55:59,639 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6362e5a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:55:59,639 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x4654132d to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:55:59,643 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6be8c20b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:55:59,644 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x00e18ba3 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:55:59,647 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46b5e40, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:55:59,648 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x1439cba9 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:55:59,651 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5fe050a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:55:59,652 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x3fed686f to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:55:59,688 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@655398b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:55:59,689 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x6a7ef79d to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:55:59,696 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13112432, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:55:59,697 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x6ab9467e to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:55:59,707 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@776686ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:55:59,713 DEBUG [hconnection-0x6903a248-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:55:59,714 DEBUG [hconnection-0x32e7e234-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:55:59,722 DEBUG [hconnection-0x759f09fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:55:59,722 DEBUG [hconnection-0x43d8c51a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:55:59,722 DEBUG [hconnection-0x270bee6b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:55:59,723 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49494, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:55:59,724 DEBUG [hconnection-0x7a579215-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:55:59,724 DEBUG [hconnection-0x77cf76aa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:55:59,725 DEBUG [hconnection-0x58c2494-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:55:59,730 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49480, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:55:59,731 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49500, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:55:59,736 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49524, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:55:59,740 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49554, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:55:59,735 DEBUG [hconnection-0x709f9e55-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:55:59,746 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49510, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:55:59,747 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49552, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:55:59,743 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49536, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:55:59,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:55:59,770 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49564, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:55:59,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:55:59,791 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:55:59,792 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:55:59,792 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:55:59,792 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:55:59,794 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:55:59,794 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:55:59,796 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:55:59,796 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:55:59,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:55:59,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:55:59,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:55:59,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:55:59,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:55:59,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:55:59,814 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:55:59,814 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:55:59,830 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:55:59,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:55:59,830 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:55:59,858 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:55:59,858 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-27 04:55:59,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:55:59,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:55:59,872 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:55:59,873 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:55:59,873 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:55:59,874 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:55:59,875 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:55:59,876 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:55:59,876 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,876 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:55:59,876 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:55:59,883 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:55:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:55:59,887 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:55:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:55:59,887 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:55:59,887 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:55:59,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:55:59,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:55:59,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:55:59,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:55:59,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c12a790[Count = 0] remaining members to acquire global barrier 2023-03-27 04:55:59,890 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:55:59,890 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:55:59,892 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:55:59,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:55:59,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:55:59,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:55:59,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:55:59,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:55:59,898 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:55:59,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:55:59,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:55:59,898 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:55:59,898 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:55:59,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:55:59,899 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,899 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:55:59,908 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:55:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:55:59,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:55:59,912 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:55:59,913 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:55:59,913 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:55:59,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:55:59,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:55:59,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:55:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:55:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:55:59,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:55:59,925 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:55:59,925 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:55:59,925 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:55:59,925 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:55:59,925 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:55:59,963 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:55:59,963 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:55:59,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:55:59,963 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:55:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:55:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:55:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:55:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:55:59,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:55:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:55:59,965 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:55:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:55:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:55:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:55:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:55:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,969 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:55:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:55:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,980 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:55:59,980 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:55:59,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:55:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:55:59,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:55:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:55:59,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:55:59,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:55:59,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:55:59,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:55:59,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:55:59,981 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:55:59,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:55:59,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:55:59,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:00,012 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:00,014 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:00,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893020011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893020022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893020015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893020025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893020027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,114 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:00,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:00,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:00,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893020132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:00,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:00,136 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:00,136 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:00,137 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:00,137 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:00,138 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,138 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,139 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:00,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,142 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:00,143 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,143 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:00,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893020147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893020147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893020148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893020147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:00,152 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:00,152 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:00,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:00,159 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:00,159 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:00,160 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:00,160 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:00,160 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:00,160 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:00,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:00,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:00,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:00,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:00,164 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:00,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:00,168 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:00,168 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,168 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:00,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:00,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa56233[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:00,171 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:00,171 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,178 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,178 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:00,178 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:00,179 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:00,179 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,179 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:00,183 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:00,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:00,183 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:00,183 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:00,183 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:00,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:00,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:00,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:00,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:00,190 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:00,190 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:00,190 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:00,190 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:00,190 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:00,196 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,196 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,196 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:00,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:00,197 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:00,199 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,213 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:00,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:00,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:00,213 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,213 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,214 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,214 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,214 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,214 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,214 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,216 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:00,217 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:00,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:00,217 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:00,217 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:00,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:00,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:00,317 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:00,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:00,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:00,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:00,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:00,328 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:00,328 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:00,342 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:00,342 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:00,342 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,342 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,343 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:00,343 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:00,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:00,344 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,344 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:00,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:00,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:00,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893020343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:00,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:00,349 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:00,349 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:00,350 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:00,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:00,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:00,352 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:00,352 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:00,352 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:00,352 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:00,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:00,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:00,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,354 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:00,354 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:00,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893020354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893020356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893020358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893020358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,365 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:00,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:00,366 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:00,366 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:00,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:00,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@445cf515[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:00,368 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:00,369 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,373 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,373 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:00,373 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:00,373 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:00,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:00,375 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:00,375 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,376 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:00,376 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:00,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:00,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:00,383 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:00,383 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:00,384 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:00,384 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:00,384 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:00,390 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,390 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:00,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,390 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:00,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:00,392 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,395 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,404 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,404 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,404 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,404 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:00,404 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:00,404 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:00,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:00,404 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:00,404 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:00,405 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,405 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:00,405 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:00,405 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:00,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:00,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/140dc831fe7440d4aa0471f4933bbd28 2023-03-27 04:56:00,513 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:00,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:00,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:00,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:00,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:00,524 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:00,524 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:00,526 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:00,526 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:00,526 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,526 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,527 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:00,527 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,528 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,528 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:00,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:00,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:00,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:00,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:00,532 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:00,532 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:00,533 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:00,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:00,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:00,535 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:00,535 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:00,535 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:00,535 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:00,538 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:00,538 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:00,538 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,538 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:00,539 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:00,550 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,550 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:00,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:00,551 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,551 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:00,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:00,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@337d8857[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:00,556 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:00,556 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,561 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,562 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:00,562 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:00,562 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:00,562 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,562 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:00,567 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:00,567 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:00,567 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:00,567 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:00,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:00,571 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:00,571 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:00,571 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:00,571 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:00,571 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:00,578 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,578 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:00,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,578 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:00,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:00,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:00,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:00,583 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,584 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,592 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:00,593 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:00,593 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:00,593 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,593 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:00,593 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,593 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,593 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,593 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,593 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:00,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:00,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:00,594 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:00,594 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:00,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:00,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893020653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893020667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893020667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,670 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/6ac434dd19104755959a879710ce83e7 2023-03-27 04:56:00,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893020669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:00,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893020670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,694 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:00,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:00,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:00,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:00,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:00,722 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:00,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:00,735 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:00,736 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:00,736 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,736 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,739 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:00,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:00,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:00,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,741 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:00,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:00,742 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,742 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:00,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:00,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:00,747 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:00,749 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:00,749 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:00,750 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:00,750 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:00,750 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:00,751 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:00,752 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:00,752 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:00,752 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:00,753 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:00,753 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,753 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:00,753 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:00,756 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:00,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:00,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:00,758 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:00,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a1a878f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:00,758 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,760 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,760 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:00,760 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:00,760 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:00,762 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,762 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:00,762 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:00,762 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:00,762 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:00,762 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:00,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:00,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:00,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:00,777 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:00,777 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:00,777 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:00,777 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:00,777 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:00,788 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,788 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,788 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:00,789 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,791 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:00,799 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,799 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,799 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,799 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,799 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,799 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,799 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,799 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:00,799 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:00,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:00,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:00,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:00,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:00,799 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:00,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:00,801 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:00,801 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:00,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:00,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:00,901 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:00,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:00,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:00,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:00,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:00,912 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:00,912 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:00,914 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:00,915 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:00,915 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,915 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,916 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:00,916 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:00,917 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,917 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:00,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:00,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:00,919 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:00,919 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:00,920 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:00,920 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:00,921 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:00,922 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:00,922 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:00,922 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:00,922 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:00,931 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:00,931 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:00,931 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,931 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:00,931 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:00,936 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,936 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:00,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:00,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:00,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:00,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:00,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6ac605[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:00,939 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:00,940 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,941 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,941 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:00,941 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,941 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:00,941 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:00,941 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:00,943 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:00,943 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:00,943 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:00,943 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:00,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:00,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:00,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:00,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:00,952 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:00,952 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:00,952 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:00,952 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:00,952 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:00,957 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,957 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,957 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:00,957 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:00,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:00,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,966 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:00,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:00,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:00,970 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:00,970 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:00,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:00,970 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:00,970 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:00,970 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,970 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:00,970 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:00,970 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,970 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:00,970 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:00,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:00,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:00,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:00,971 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:00,971 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:00,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:00,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:01,071 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:01,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:01,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:01,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:01,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:01,078 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:01,078 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:01,079 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:01,079 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:01,079 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,079 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,081 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,081 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:01,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:01,081 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,081 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:01,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:01,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:01,084 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:01,084 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:01,085 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:01,094 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:01,094 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:01,094 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:01,094 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:01,094 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:01,095 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:01,096 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:01,096 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:01,096 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,096 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:01,096 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:01,098 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,098 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:01,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:01,099 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,099 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:01,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:01,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:01,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:01,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:01,101 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:01,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d3d7880[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:01,101 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,102 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,103 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:01,103 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:01,103 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:01,103 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,103 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:01,104 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,104 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:01,104 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:01,104 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:01,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:01,110 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:01,110 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:01,110 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:01,110 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:01,110 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:01,115 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,115 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:01,115 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:01,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:01,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:01,116 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,120 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,128 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:01,128 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:01,128 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,128 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,129 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,129 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,129 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:01,129 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:01,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:01,129 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:01,129 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,129 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:01,131 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:01,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:01,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:01,153 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/bfa559db3dce45169e963f3c5cfbdca8 2023-03-27 04:56:01,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:01,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893021166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,168 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/140dc831fe7440d4aa0471f4933bbd28 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/140dc831fe7440d4aa0471f4933bbd28 2023-03-27 04:56:01,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:01,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893021174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:01,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893021176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:01,184 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/140dc831fe7440d4aa0471f4933bbd28, entries=200, sequenceid=14, filesize=13.9 K 2023-03-27 04:56:01,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893021181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,188 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-27 04:56:01,191 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/6ac434dd19104755959a879710ce83e7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/6ac434dd19104755959a879710ce83e7 2023-03-27 04:56:01,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/6ac434dd19104755959a879710ce83e7, entries=150, sequenceid=14, filesize=11.6 K 2023-03-27 04:56:01,230 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/bfa559db3dce45169e963f3c5cfbdca8 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bfa559db3dce45169e963f3c5cfbdca8 2023-03-27 04:56:01,231 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:01,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:01,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:01,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:01,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:01,240 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:01,240 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:01,241 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:01,241 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:01,242 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,242 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,243 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,243 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,243 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:01,244 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:01,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:01,244 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bfa559db3dce45169e963f3c5cfbdca8, entries=150, sequenceid=14, filesize=11.6 K 2023-03-27 04:56:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:01,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:01,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:01,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:01,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:01,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:01,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:01,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:01,253 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for e7756c618a32a862ad613327bee74d79 in 1390ms, sequenceid=14, compaction requested=false 2023-03-27 04:56:01,253 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:01,253 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:01,253 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:01,254 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:01,255 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:01,255 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:01,255 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,255 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:01,255 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:01,256 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-27 04:56:01,256 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,256 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:01,257 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,257 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:01,259 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:01,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:01,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42dd7440[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:01,259 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:01,259 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,260 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,260 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:01,260 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:01,260 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:01,261 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,261 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:01,262 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:01,262 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:01,262 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:01,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:01,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:01,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:01,268 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:01,268 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:01,268 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:01,268 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:01,268 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:01,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:01,271 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-27 04:56:01,271 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:01,271 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:01,272 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:01,272 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:01,272 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:01,272 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:01,274 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:01,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:01,274 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:01,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:01,291 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,291 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:01,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,297 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,301 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:01,301 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:01,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:01,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:01,302 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,302 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,302 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,303 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,303 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,303 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:01,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:01,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:01,303 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:01,303 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:01,303 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,303 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:01,404 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:01,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:01,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:01,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:01,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:01,416 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:01,416 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:01,417 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:01,417 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:01,418 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,418 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,420 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:01,420 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:01,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:01,421 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,421 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:01,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:01,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:01,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:01,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:01,423 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:01,423 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:01,424 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:01,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:01,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:01,427 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:01,428 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:01,428 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:01,428 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:01,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:01,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:01,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:01,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:01,445 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,445 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:01,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:01,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:01,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:01,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,449 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,449 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:01,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:01,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:01,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c55b56b[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:01,450 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:01,450 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,453 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,453 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:01,453 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:01,453 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:01,454 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,454 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:01,455 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:01,455 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:01,455 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:01,462 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:01,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:01,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:01,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:01,466 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:01,466 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:01,466 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:01,466 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:01,466 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:01,470 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,470 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:01,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:01,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:01,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,472 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,473 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:01,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:01,474 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:01,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:01,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,482 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,482 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:01,482 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,482 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,482 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,482 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,482 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,482 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:01,482 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:01,482 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:01,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:01,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:01,488 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:01,488 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:01,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:01,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:01,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:01,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893021527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,588 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:01,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:01,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:01,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:01,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:01,595 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:01,595 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:01,599 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:01,599 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:01,600 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,600 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,601 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,601 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:01,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:01,601 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,601 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:01,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:01,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:01,605 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:01,605 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:01,608 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:01,609 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:01,609 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:01,622 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:01,622 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:01,622 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:01,622 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:01,630 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:01,630 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:01,630 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,630 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:01,630 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:01,632 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,632 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:01,632 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,632 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:01,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:01,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:01,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:01,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:01,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e70c0a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:01,634 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:01,634 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,638 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,638 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:01,638 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:01,638 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:01,638 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,639 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:01,639 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:01,640 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:01,640 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,640 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:01,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:01,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:01,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:01,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:01,648 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:01,648 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:01,648 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:01,648 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:01,648 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:01,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:01,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893021646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,658 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,658 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:01,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,659 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:01,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:01,659 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,662 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,670 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:01,670 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:01,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:01,673 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,673 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,673 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,673 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:01,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:01,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:01,673 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,673 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,673 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,673 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:01,673 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:01,673 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:01,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:01,774 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:01,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:01,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:01,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:01,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:01,787 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:01,787 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:01,788 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:01,788 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:01,788 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,788 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,789 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,789 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:01,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:01,790 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,790 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:01,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:01,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:01,793 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:01,793 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:01,805 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:01,805 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/26c637eefad242bca77883e69e6da2a7 2023-03-27 04:56:01,815 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:01,815 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:01,815 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:01,815 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:01,815 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:01,815 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:01,835 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:01,835 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:01,835 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,835 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:01,835 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:01,837 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,837 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,837 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,837 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:01,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:01,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:01,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:01,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72e89d6e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:01,841 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:01,841 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,841 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,842 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:01,842 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:01,842 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:01,842 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,842 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:01,843 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:01,843 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,843 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:01,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:01,843 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:01,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:01,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:01,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:01,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:01,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:01,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:01,858 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:01,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:01,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:01,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:01,858 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:01,860 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,860 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:01,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,860 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:01,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:01,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:01,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:01,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:01,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:01,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:01,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:01,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893021860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:01,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:01,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:01,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:01,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:01,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:01,865 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:01,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:01,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:01,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:01,866 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,879 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,879 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,879 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:01,879 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,881 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:01,881 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:01,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:01,981 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:01,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:01,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:01,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:01,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:01,988 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:01,988 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:01,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:01,991 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:01,991 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,991 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,994 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:01,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:01,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:01,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:01,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:01,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:01,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:01,998 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:01,998 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:02,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:02,003 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:02,003 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:02,004 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:02,005 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:02,005 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:02,005 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:02,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:02,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:02,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:02,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:02,012 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,012 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:02,012 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,012 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:02,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:02,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:02,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:02,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bbe9807[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:02,015 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:02,015 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,017 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:02,017 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:02,017 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:02,018 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,018 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:02,020 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:02,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:02,021 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:02,021 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:02,021 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:02,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:02,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:02,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:02,027 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:02,027 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:02,027 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:02,027 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:02,027 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:02,040 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,040 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:02,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,040 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:02,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:02,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:02,041 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:02,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:02,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:02,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,043 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,048 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,048 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:02,048 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:02,048 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:02,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:02,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:02,048 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:02,048 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:02,048 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,050 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,050 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,050 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:02,050 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:02,050 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:02,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:02,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:02,151 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:02,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:02,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:02,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:02,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:02,157 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:02,157 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:02,163 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:02,163 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:02,163 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,163 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,165 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:02,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893022165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,166 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:02,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:02,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:02,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,167 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,168 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:02,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:02,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:02,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:02,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893022176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:02,186 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:02,186 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:02,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:02,187 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:02,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893022182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:02,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893022184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,194 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:02,194 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:02,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:02,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893022190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,195 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:02,195 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:02,195 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:02,195 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:02,196 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:02,197 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:02,197 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,197 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:02,197 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:02,199 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,199 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,199 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:02,200 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:02,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:02,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:02,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:02,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a8180f6[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:02,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:02,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,206 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,206 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,207 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:02,207 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:02,207 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:02,207 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:02,208 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:02,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:02,209 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:02,210 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:02,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:02,210 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:02,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:02,228 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:02,228 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:02,228 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:02,228 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:02,228 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:02,230 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,230 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:02,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,230 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:02,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:02,232 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,233 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:02,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,236 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:02,237 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:02,237 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:02,237 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,239 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,239 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,239 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,239 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,239 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,239 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:02,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:02,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:02,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:02,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:02,240 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:02,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:02,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:02,240 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:02,340 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:02,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:02,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:02,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:02,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:02,345 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:02,345 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:02,346 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:02,346 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:02,347 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,347 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,348 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,348 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,348 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:02,349 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:02,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:02,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:02,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:02,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:02,351 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:02,351 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:02,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:02,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:02,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:02,352 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:02,353 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:02,353 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:02,353 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:02,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:02,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:02,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,354 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:02,354 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:02,357 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/935ce82e8afc403c82c7d4afa20b7151 2023-03-27 04:56:02,361 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,361 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:02,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:02,362 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:02,362 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:02,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:02,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:02,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26ba48f0[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:02,368 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:02,368 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,373 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,373 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,373 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:02,373 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:02,374 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:02,374 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:02,379 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:02,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,379 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:02,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:02,379 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:02,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,389 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/bbef6359fbd94d989716595a3370ba35 2023-03-27 04:56:02,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:02,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:02,390 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:02,390 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:02,390 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:02,390 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:02,390 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:02,396 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,396 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:02,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,396 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:02,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:02,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:02,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:02,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,397 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:02,400 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,403 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,403 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,403 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:02,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:02,403 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,404 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,404 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,404 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,403 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:02,404 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:02,404 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:02,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:02,405 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:02,406 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:02,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:02,407 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/26c637eefad242bca77883e69e6da2a7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/26c637eefad242bca77883e69e6da2a7 2023-03-27 04:56:02,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:02,424 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/26c637eefad242bca77883e69e6da2a7, entries=200, sequenceid=34, filesize=13.9 K 2023-03-27 04:56:02,428 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/935ce82e8afc403c82c7d4afa20b7151 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/935ce82e8afc403c82c7d4afa20b7151 2023-03-27 04:56:02,448 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/935ce82e8afc403c82c7d4afa20b7151, entries=150, sequenceid=34, filesize=11.6 K 2023-03-27 04:56:02,456 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/bbef6359fbd94d989716595a3370ba35 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bbef6359fbd94d989716595a3370ba35 2023-03-27 04:56:02,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bbef6359fbd94d989716595a3370ba35, entries=150, sequenceid=34, filesize=11.6 K 2023-03-27 04:56:02,479 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e7756c618a32a862ad613327bee74d79 in 1207ms, sequenceid=34, compaction requested=false 2023-03-27 04:56:02,479 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:02,504 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:02,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:02,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:02,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:02,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:02,510 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:02,510 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:02,511 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:02,511 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:02,512 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,512 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,514 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,515 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,515 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:02,516 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:02,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:02,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:02,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:02,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:02,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:02,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:02,520 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:02,520 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:02,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:02,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:02,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:02,521 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:02,521 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:02,521 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-27 04:56:02,521 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:02,522 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:02,522 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:02,522 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:02,522 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:02,522 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:02,567 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/662fc6e3873141ce9b973e562a247c73 2023-03-27 04:56:02,620 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/ab847b59941747fdbdf60f14f1e92094 2023-03-27 04:56:02,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:02,679 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:02,691 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/52b6b6ff5fef471692c67d51fadc60dd 2023-03-27 04:56:02,703 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/662fc6e3873141ce9b973e562a247c73 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/662fc6e3873141ce9b973e562a247c73 2023-03-27 04:56:02,717 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/662fc6e3873141ce9b973e562a247c73, entries=150, sequenceid=46, filesize=11.6 K 2023-03-27 04:56:02,720 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/ab847b59941747fdbdf60f14f1e92094 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/ab847b59941747fdbdf60f14f1e92094 2023-03-27 04:56:02,733 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/ab847b59941747fdbdf60f14f1e92094, entries=150, sequenceid=46, filesize=11.6 K 2023-03-27 04:56:02,735 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/52b6b6ff5fef471692c67d51fadc60dd as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/52b6b6ff5fef471692c67d51fadc60dd 2023-03-27 04:56:02,753 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/52b6b6ff5fef471692c67d51fadc60dd, entries=150, sequenceid=46, filesize=11.6 K 2023-03-27 04:56:02,756 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=40.25 KB/41220 for e7756c618a32a862ad613327bee74d79 in 234ms, sequenceid=46, compaction requested=true 2023-03-27 04:56:02,757 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:02,757 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:02,757 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:02,767 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:02,767 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,767 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:02,767 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:02,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:02,768 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:56:02,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,768 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:02,768 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:02,769 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:02,769 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:02,769 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:02,769 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:02,769 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:02,770 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,771 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:02,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:02,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@108fa2a4[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:02,777 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:02,777 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,783 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,783 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:02,783 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:02,783 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:02,784 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,784 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:02,786 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:02,786 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:02,786 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:02,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:02,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:02,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:02,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:02,801 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:02,801 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:02,801 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:02,801 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:02,801 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:02,803 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,803 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:02,803 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:02,803 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,805 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,810 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:02,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:02,810 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,810 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:02,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:02,811 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:02,811 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,811 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,811 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,811 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:02,811 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:02,811 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:02,812 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:02,812 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:02,912 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:02,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:02,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:02,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:02,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:02,930 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:02,930 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:02,931 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:02,931 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:02,934 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,934 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,935 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:02,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:02,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:02,936 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:02,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:02,943 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,943 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:02,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:02,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:02,943 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:02,943 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:02,944 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:02,944 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:02,944 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:02,944 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:02,944 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:02,944 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:02,945 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:02,945 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:02,945 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:02,945 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,945 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:02,946 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:02,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:02,948 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,949 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,949 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:02,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:02,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f5e7e12[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:02,958 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:02,959 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,961 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,961 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,962 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:02,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,962 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:02,962 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:02,962 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:02,963 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,963 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:02,963 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:02,963 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:02,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:02,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:02,973 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:02,973 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:02,973 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:02,973 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:02,973 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:02,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:02,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:02,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:02,979 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,981 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,987 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-27 04:56:02,988 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:02,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:02,988 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,988 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,988 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:02,989 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:02,989 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,989 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:02,989 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:02,989 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:02,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:02,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:02,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:02,989 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:02,989 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:02,989 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:02,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:03,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679892962926, max timestamp=1679892962996Num uniques 859; 2023-03-27 04:56:03,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:03,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679892962926, max timestamp=1679892962996Num uniques 859; 2023-03-27 04:56:03,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-27 04:56:03,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679892962926, max timestamp=1679892962996Num uniques 859; 2023-03-27 04:56:03,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-27 04:56:03,089 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:03,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:03,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:03,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:03,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:03,099 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:03,099 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:03,100 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:03,100 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:03,101 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,101 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,102 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,102 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:03,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:03,103 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,103 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:03,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:03,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:03,113 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:03,114 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:03,115 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:03,117 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:03,118 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:03,119 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:03,119 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:03,119 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:03,121 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:03,121 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:03,121 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:03,122 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,122 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:03,122 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:03,123 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,123 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:03,124 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,124 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:03,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:03,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:03,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:03,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@294edf3f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:03,126 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:03,127 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,128 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,128 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:03,128 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:03,128 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:03,128 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,128 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:03,129 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,129 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:03,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:03,129 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:03,130 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:03,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:03,147 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:03,147 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:03,147 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:03,147 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:03,147 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:03,149 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,149 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,149 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:03,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:03,150 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,152 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,161 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:03,162 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:03,162 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,162 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:03,162 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:03,162 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,162 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,162 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,163 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:03,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:03,162 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:03,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:03,163 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:03,167 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:03,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:03,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:03,172 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-27 04:56:03,174 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-27 04:56:03,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:03,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893023216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,267 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:03,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:03,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:03,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:03,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:03,274 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:03,274 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:03,275 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:03,275 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:03,276 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,277 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,277 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=56 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/c0247ebceddb41e1b04f2bb3fd343dc8 2023-03-27 04:56:03,278 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:03,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:03,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:03,279 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,283 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,283 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:03,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:03,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:03,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:03,289 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:03,289 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:03,293 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:03,293 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:03,293 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:03,293 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:03,293 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:03,294 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:03,294 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:03,294 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:03,294 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:03,295 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,295 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:03,295 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:03,296 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:03,297 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:03,299 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,299 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:03,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:03,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:03,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:03,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d47ef2e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:03,318 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:03,318 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,320 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,320 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:03,322 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,322 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:03,322 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:03,322 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:03,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:03,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893023321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,325 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:03,325 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,325 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:03,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:03,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:03,326 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:03,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:03,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:03,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:03,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:03,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:03,329 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:03,330 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:03,330 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:03,330 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:03,330 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:03,331 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,331 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:03,331 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:03,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:03,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:03,332 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:03,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:03,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:03,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,375 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,375 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:03,375 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:03,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:03,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:03,375 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:03,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:03,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:03,378 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:03,378 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:03,375 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,378 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:03,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:03,478 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:03,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:03,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:03,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:03,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:03,485 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:03,485 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:03,488 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:03,488 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:03,488 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,489 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,490 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:03,490 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:03,491 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,491 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:03,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:03,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:03,493 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:03,493 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:03,493 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:03,494 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:03,494 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:03,494 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:03,494 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:03,494 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:03,494 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:03,494 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:03,494 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:03,494 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,494 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:03,494 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:03,496 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,496 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:03,496 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,496 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:03,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:03,498 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:03,498 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30b0eca0[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:03,498 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,499 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:03,499 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:03,499 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:03,500 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,500 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:03,501 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,501 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:03,501 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:03,501 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:03,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:03,504 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:03,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:03,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:03,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:03,505 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:03,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:03,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:03,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:03,507 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,508 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,510 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:03,510 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:03,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:03,510 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,510 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:03,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:03,510 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,511 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,511 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,511 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,511 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,510 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:03,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:03,511 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:03,514 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:03,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:03,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:03,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893023528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,615 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:03,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:03,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:03,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:03,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:03,621 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:03,621 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:03,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:03,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:03,624 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,624 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,625 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,625 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:03,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:03,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:03,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:03,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:03,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:03,627 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:03,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:03,627 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:03,627 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:03,630 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:03,631 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:03,631 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:03,631 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:03,631 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:03,631 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:03,631 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:03,631 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:03,632 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:03,632 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,632 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:03,632 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:03,640 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,640 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:03,640 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,640 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:03,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:03,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:03,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:03,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c52ba4a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:03,642 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:03,642 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,645 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,645 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:03,645 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:03,645 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:03,645 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:03,646 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:03,647 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:03,647 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:03,672 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:03,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:03,680 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:03,680 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:03,681 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:03,681 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:03,681 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:03,682 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,682 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:03,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,682 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:03,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:03,684 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:03,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,685 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:03,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,689 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:03,689 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:03,689 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,689 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:03,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:03,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:03,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:03,689 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:03,689 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:03,689 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:03,689 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:03,689 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,700 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,700 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,700 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:03,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:03,771 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=56 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/e06010bd47234d3fa6153d310d495cf9 2023-03-27 04:56:03,790 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:03,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:03,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:03,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:03,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:03,802 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:03,802 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:03,802 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:03,802 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:03,803 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,803 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,804 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,805 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,805 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:03,806 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:03,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:03,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:03,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:03,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:03,839 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:03,839 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:03,839 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:03,841 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:03,841 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:03,841 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:03,841 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:03,841 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:03,841 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:03,842 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:03,842 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:03,842 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,842 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:03,842 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:03,844 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,844 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,844 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:03,845 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:03,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:03,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:03,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893023841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:03,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:03,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23ba35d8[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:03,852 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:03,852 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,852 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=56 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/f8b3d0d0cf344ca796f4df837b0ecce0 2023-03-27 04:56:03,856 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:03,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:03,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:03,861 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:03,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,861 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:03,861 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:03,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:03,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:03,863 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:03,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:03,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:03,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:03,863 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:03,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:03,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:03,866 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:03,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,868 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:03,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,871 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:03,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:03,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:03,871 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:03,872 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,872 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,872 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:03,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:03,871 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:03,872 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:03,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:03,873 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:03,873 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:03,873 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:03,873 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:03,873 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:03,876 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/c0247ebceddb41e1b04f2bb3fd343dc8 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c0247ebceddb41e1b04f2bb3fd343dc8 2023-03-27 04:56:03,888 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c0247ebceddb41e1b04f2bb3fd343dc8, entries=150, sequenceid=56, filesize=11.6 K 2023-03-27 04:56:03,892 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/e06010bd47234d3fa6153d310d495cf9 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/e06010bd47234d3fa6153d310d495cf9 2023-03-27 04:56:03,902 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/e06010bd47234d3fa6153d310d495cf9, entries=100, sequenceid=56, filesize=9.3 K 2023-03-27 04:56:03,908 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/f8b3d0d0cf344ca796f4df837b0ecce0 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f8b3d0d0cf344ca796f4df837b0ecce0 2023-03-27 04:56:03,924 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f8b3d0d0cf344ca796f4df837b0ecce0, entries=100, sequenceid=56, filesize=9.3 K 2023-03-27 04:56:03,940 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=133.39 KB/136590 for e7756c618a32a862ad613327bee74d79 in 1171ms, sequenceid=56, compaction requested=true 2023-03-27 04:56:03,940 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:03,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:03,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:03,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:03,959 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:56:03,966 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:56:03,973 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:03,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:03,975 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:56:03,977 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/B is initiating minor compaction (all files) 2023-03-27 04:56:03,977 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/B in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:03,977 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/6ac434dd19104755959a879710ce83e7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/935ce82e8afc403c82c7d4afa20b7151, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/ab847b59941747fdbdf60f14f1e92094, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/e06010bd47234d3fa6153d310d495cf9] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=44.1 K 2023-03-27 04:56:03,981 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 6ac434dd19104755959a879710ce83e7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679892959771 2023-03-27 04:56:03,983 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 935ce82e8afc403c82c7d4afa20b7151, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1679892960012 2023-03-27 04:56:03,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:03,989 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting ab847b59941747fdbdf60f14f1e92094, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1679892961470 2023-03-27 04:56:03,990 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting e06010bd47234d3fa6153d310d495cf9, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1679892962698 2023-03-27 04:56:04,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:04,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:04,001 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:04,001 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:04,001 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:04,001 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:04,001 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:04,002 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:04,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:04,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:04,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:04,003 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,003 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,003 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:04,003 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52220 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:56:04,004 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/A is initiating minor compaction (all files) 2023-03-27 04:56:04,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:04,004 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/A in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:04,004 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/140dc831fe7440d4aa0471f4933bbd28, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/26c637eefad242bca77883e69e6da2a7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/662fc6e3873141ce9b973e562a247c73, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c0247ebceddb41e1b04f2bb3fd343dc8] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=51.0 K 2023-03-27 04:56:04,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:04,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:04,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:04,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:04,005 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 140dc831fe7440d4aa0471f4933bbd28, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679892959771 2023-03-27 04:56:04,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:04,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:04,008 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:04,008 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 26c637eefad242bca77883e69e6da2a7, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1679892960012 2023-03-27 04:56:04,008 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:04,008 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:04,009 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 662fc6e3873141ce9b973e562a247c73, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1679892961470 2023-03-27 04:56:04,009 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting c0247ebceddb41e1b04f2bb3fd343dc8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1679892962698 2023-03-27 04:56:04,026 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:04,026 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:04,026 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:04,027 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=133.39 KB heapSize=383.16 KB 2023-03-27 04:56:04,027 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:04,027 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:04,028 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:04,028 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:04,028 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:04,028 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:04,026 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:04,039 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#B#compaction#12 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:04,051 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#A#compaction#14 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:04,171 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/ccf3f27ea5144588bbaad9db8cf0f76b as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/ccf3f27ea5144588bbaad9db8cf0f76b 2023-03-27 04:56:04,199 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e7756c618a32a862ad613327bee74d79/B of e7756c618a32a862ad613327bee74d79 into ccf3f27ea5144588bbaad9db8cf0f76b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:04,199 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:04,199 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/B, priority=12, startTime=1679892963959; duration=0sec 2023-03-27 04:56:04,204 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:04,205 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:56:04,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:04,210 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:04,224 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:56:04,224 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/C is initiating minor compaction (all files) 2023-03-27 04:56:04,225 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/C in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:04,225 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bfa559db3dce45169e963f3c5cfbdca8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bbef6359fbd94d989716595a3370ba35, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/52b6b6ff5fef471692c67d51fadc60dd, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f8b3d0d0cf344ca796f4df837b0ecce0] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=44.1 K 2023-03-27 04:56:04,226 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting bfa559db3dce45169e963f3c5cfbdca8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679892959771 2023-03-27 04:56:04,227 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting bbef6359fbd94d989716595a3370ba35, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1679892960012 2023-03-27 04:56:04,228 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 52b6b6ff5fef471692c67d51fadc60dd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1679892961470 2023-03-27 04:56:04,229 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting f8b3d0d0cf344ca796f4df837b0ecce0, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1679892962698 2023-03-27 04:56:04,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:04,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893024225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:04,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893024227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:04,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893024227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:04,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893024243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,274 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#C#compaction#15 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:04,352 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/a107ec6aeaf64992adb1e32a594c0a44 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/a107ec6aeaf64992adb1e32a594c0a44 2023-03-27 04:56:04,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:04,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893024347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:04,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:04,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893024348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893024349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:04,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893024349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,366 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e7756c618a32a862ad613327bee74d79/C of e7756c618a32a862ad613327bee74d79 into a107ec6aeaf64992adb1e32a594c0a44(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:04,366 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:04,366 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/C, priority=12, startTime=1679892963959; duration=0sec 2023-03-27 04:56:04,366 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:04,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:04,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893024361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:04,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893024555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:04,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893024561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:04,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893024562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,574 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/d486958c13dc47829f9475717b8ddb82 2023-03-27 04:56:04,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:04,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893024570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,588 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/f3f9256101324bab8d6b02400c53bf2f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/f3f9256101324bab8d6b02400c53bf2f 2023-03-27 04:56:04,622 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e7756c618a32a862ad613327bee74d79/A of e7756c618a32a862ad613327bee74d79 into f3f9256101324bab8d6b02400c53bf2f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:04,622 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:04,622 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/A, priority=12, startTime=1679892963942; duration=0sec 2023-03-27 04:56:04,622 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:04,663 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/461e6e18baad45cab26d014a151b1f3f 2023-03-27 04:56:04,691 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-27 04:56:04,691 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-27 04:56:04,719 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/4665b9efd869472498f9c6a7dc2ad8c2 2023-03-27 04:56:04,730 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/d486958c13dc47829f9475717b8ddb82 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d486958c13dc47829f9475717b8ddb82 2023-03-27 04:56:04,739 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d486958c13dc47829f9475717b8ddb82, entries=150, sequenceid=93, filesize=11.6 K 2023-03-27 04:56:04,741 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/461e6e18baad45cab26d014a151b1f3f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/461e6e18baad45cab26d014a151b1f3f 2023-03-27 04:56:04,764 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/461e6e18baad45cab26d014a151b1f3f, entries=150, sequenceid=93, filesize=11.6 K 2023-03-27 04:56:04,770 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/4665b9efd869472498f9c6a7dc2ad8c2 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/4665b9efd869472498f9c6a7dc2ad8c2 2023-03-27 04:56:04,784 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/4665b9efd869472498f9c6a7dc2ad8c2, entries=150, sequenceid=93, filesize=11.6 K 2023-03-27 04:56:04,788 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~133.39 KB/136590, heapSize ~383.11 KB/392304, currentSize=46.96 KB/48090 for e7756c618a32a862ad613327bee74d79 in 762ms, sequenceid=93, compaction requested=false 2023-03-27 04:56:04,788 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:04,788 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:04,788 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:04,804 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:04,804 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,804 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:04,804 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:04,805 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:04,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:04,805 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:04,806 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:04,806 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:04,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:04,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:04,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:04,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4166801e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:04,808 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:04,808 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:04,810 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:04,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:04,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:04,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:04,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:04,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:04,811 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:04,811 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:04,811 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:04,813 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,813 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,813 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:04,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:04,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:04,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:04,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:04,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:04,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:04,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:04,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:04,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:04,816 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:04,816 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:04,816 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:04,816 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:04,816 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:04,818 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:04,818 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:04,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:04,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:04,819 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:04,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:04,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:04,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:04,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:04,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:04,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:04,820 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:04,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:04,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:04,823 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:04,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:04,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:04,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,831 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,832 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:04,832 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:04,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:04,832 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:04,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:04,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:04,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:04,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:04,832 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,832 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:04,832 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:04,832 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:04,832 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:04,832 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:04,832 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:04,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:04,832 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:04,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:04,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:04,861 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-27 04:56:04,862 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:04,862 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:04,862 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:04,862 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:04,862 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:04,862 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:04,892 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/c8fc1940f8554d838d5b79ebd5127c43 2023-03-27 04:56:04,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679892964900, max timestamp=1679892964914Num uniques 859; 2023-03-27 04:56:04,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:04,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679892964900, max timestamp=1679892964914Num uniques 859; 2023-03-27 04:56:04,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:04,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679892964900, max timestamp=1679892964915Num uniques 859; 2023-03-27 04:56:04,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:04,932 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:04,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:04,944 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/11f723e8a88a4d3f9256d358ac156ba8 2023-03-27 04:56:04,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:04,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:04,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:04,950 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:04,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:04,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:04,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:04,951 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:04,951 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:04,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:04,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:04,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:04,952 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,953 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,953 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:04,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:04,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:04,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:04,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:04,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:04,955 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:04,955 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:04,955 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:04,955 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:04,956 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:04,956 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:04,956 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:04,956 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:04,956 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:04,956 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:04,956 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:04,956 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,956 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:04,956 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:04,959 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:04,959 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:04,959 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:04,961 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:04,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:04,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:04,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:04,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:04,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7684b17e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:04,964 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:04,964 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:04,975 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:04,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:04,975 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,975 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:04,975 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:04,975 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:04,975 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:04,978 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:04,978 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:04,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,978 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:04,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:04,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:04,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:04,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:04,981 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/e01a3ce1cb584df3898c04eb8a3e8735 2023-03-27 04:56:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:04,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:04,982 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:04,982 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:04,982 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:04,982 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:04,983 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:04,990 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:04,990 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:04,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:04,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:04,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:04,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:04,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:04,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893024979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:04,991 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:04,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:04,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:04,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:04,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:04,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:04,993 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/c8fc1940f8554d838d5b79ebd5127c43 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c8fc1940f8554d838d5b79ebd5127c43 2023-03-27 04:56:04,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:04,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893024987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:04,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893024991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:04,997 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893024992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,000 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:05,000 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:05,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:05,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:05,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:05,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:05,001 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,002 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:05,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:05,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:05,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,002 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:05,005 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:05,009 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c8fc1940f8554d838d5b79ebd5127c43, entries=200, sequenceid=107, filesize=13.9 K 2023-03-27 04:56:05,016 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/11f723e8a88a4d3f9256d358ac156ba8 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/11f723e8a88a4d3f9256d358ac156ba8 2023-03-27 04:56:05,027 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/11f723e8a88a4d3f9256d358ac156ba8, entries=150, sequenceid=107, filesize=11.6 K 2023-03-27 04:56:05,040 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/e01a3ce1cb584df3898c04eb8a3e8735 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/e01a3ce1cb584df3898c04eb8a3e8735 2023-03-27 04:56:05,054 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/e01a3ce1cb584df3898c04eb8a3e8735, entries=150, sequenceid=107, filesize=11.6 K 2023-03-27 04:56:05,057 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e7756c618a32a862ad613327bee74d79 in 196ms, sequenceid=107, compaction requested=true 2023-03-27 04:56:05,057 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:05,057 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:05,057 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:05,057 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:05,057 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:05,057 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:05,062 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38136 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:05,062 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/A is initiating minor compaction (all files) 2023-03-27 04:56:05,062 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/A in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:05,063 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/f3f9256101324bab8d6b02400c53bf2f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d486958c13dc47829f9475717b8ddb82, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c8fc1940f8554d838d5b79ebd5127c43] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=37.2 K 2023-03-27 04:56:05,063 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:05,063 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/B is initiating minor compaction (all files) 2023-03-27 04:56:05,063 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/B in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:05,063 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting f3f9256101324bab8d6b02400c53bf2f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1679892961501 2023-03-27 04:56:05,071 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/ccf3f27ea5144588bbaad9db8cf0f76b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/461e6e18baad45cab26d014a151b1f3f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/11f723e8a88a4d3f9256d358ac156ba8] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=35.0 K 2023-03-27 04:56:05,072 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting ccf3f27ea5144588bbaad9db8cf0f76b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1679892961501 2023-03-27 04:56:05,072 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting d486958c13dc47829f9475717b8ddb82, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1679892963189 2023-03-27 04:56:05,074 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 461e6e18baad45cab26d014a151b1f3f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1679892963189 2023-03-27 04:56:05,075 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 11f723e8a88a4d3f9256d358ac156ba8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1679892964225 2023-03-27 04:56:05,075 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting c8fc1940f8554d838d5b79ebd5127c43, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1679892964217 2023-03-27 04:56:05,104 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#B#compaction#21 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:05,106 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:05,112 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#A#compaction#22 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:05,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:05,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:05,115 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=134.18 KB heapSize=385.22 KB 2023-03-27 04:56:05,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:05,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679892964972, max timestamp=1679892965105Num uniques 1018; 2023-03-27 04:56:05,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-27 04:56:05,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679892964972, max timestamp=1679892965105Num uniques 1018; 2023-03-27 04:56:05,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-27 04:56:05,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:05,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:05,132 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:05,133 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:05,134 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:05,135 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:05,136 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,136 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679892964972, max timestamp=1679892965115Num uniques 1018; 2023-03-27 04:56:05,137 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-27 04:56:05,137 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,137 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:05,138 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:05,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:05,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:05,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,140 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:05,140 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:05,140 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:05,140 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:05,140 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:05,140 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:05,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:05,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:05,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:05,146 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:05,147 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:05,147 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:05,149 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:05,149 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:05,150 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:05,150 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:05,150 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:05,150 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:05,150 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:05,150 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:05,150 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,150 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:05,150 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:05,157 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,157 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:05,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:05,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:05,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f6dd50e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:05,162 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:05,163 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,171 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,172 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,172 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:05,173 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:05,173 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:05,173 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:05,175 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/d914c443343745ab8b2f769120fe59b0 2023-03-27 04:56:05,178 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:05,178 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:05,178 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:05,178 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:05,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:05,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:05,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:05,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:05,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:05,188 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:05,188 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:05,188 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:05,188 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:05,188 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:05,195 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:05,195 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,195 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:05,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:05,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:05,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,199 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:05,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:05,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,203 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:05,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679892965180, max timestamp=1679892965203Num uniques 859; 2023-03-27 04:56:05,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:05,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679892965180, max timestamp=1679892965194Num uniques 859; 2023-03-27 04:56:05,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:05,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679892965180, max timestamp=1679892965194Num uniques 859; 2023-03-27 04:56:05,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-27 04:56:05,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:05,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:05,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:05,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:05,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:05,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:05,228 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:05,228 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:05,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:05,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,231 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,231 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,228 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:05,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:05,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:05,243 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/9aa15ebb97f044dab136a04fb6eab2e1 2023-03-27 04:56:05,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1679892965183, max timestamp=1679892965276Num uniques 1009; 2023-03-27 04:56:05,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=16848, suffix heap size=132904,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=368 2023-03-27 04:56:05,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679892965266, max timestamp=1679892965279Num uniques 1018; 2023-03-27 04:56:05,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-27 04:56:05,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679892965266, max timestamp=1679892965276Num uniques 1018; 2023-03-27 04:56:05,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-27 04:56:05,326 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/16ded63a4ec746d985aa1ad4f2094bfb 2023-03-27 04:56:05,331 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:05,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:05,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:05,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:05,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:05,341 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:05,341 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:05,341 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:05,341 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:05,342 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,342 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,343 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:05,343 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:05,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:05,343 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,343 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:05,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:05,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:05,348 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/d914c443343745ab8b2f769120fe59b0 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d914c443343745ab8b2f769120fe59b0 2023-03-27 04:56:05,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:05,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893025334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:05,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:05,356 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:05,356 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:05,356 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:05,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:05,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893025337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679892965324, max timestamp=1679892965343Num uniques 1227; 2023-03-27 04:56:05,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=8102, suffix heap size=155576,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=177 2023-03-27 04:56:05,358 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d914c443343745ab8b2f769120fe59b0, entries=200, sequenceid=146, filesize=14.1 K 2023-03-27 04:56:05,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679892965324, max timestamp=1679892965349Num uniques 1027; 2023-03-27 04:56:05,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-03-27 04:56:05,360 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/9aa15ebb97f044dab136a04fb6eab2e1 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/9aa15ebb97f044dab136a04fb6eab2e1 2023-03-27 04:56:05,361 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:05,362 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:05,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679892965324, max timestamp=1679892965349Num uniques 1027; 2023-03-27 04:56:05,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-03-27 04:56:05,363 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:05,363 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:05,364 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:05,364 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:05,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:05,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:05,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:05,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:05,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:05,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:05,366 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:05,372 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,372 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:05,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:05,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:05,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:05,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d0b37ce[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:05,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:05,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,378 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/9aa15ebb97f044dab136a04fb6eab2e1, entries=150, sequenceid=146, filesize=11.8 K 2023-03-27 04:56:05,380 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/16ded63a4ec746d985aa1ad4f2094bfb as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/16ded63a4ec746d985aa1ad4f2094bfb 2023-03-27 04:56:05,381 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,382 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:05,382 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:05,382 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:05,382 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,382 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:05,386 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,386 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:05,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:05,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:05,386 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:05,387 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:05,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:05,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:05,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:05,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:05,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:05,390 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:05,390 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:05,390 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:05,390 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:05,390 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:05,392 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/16ded63a4ec746d985aa1ad4f2094bfb, entries=150, sequenceid=146, filesize=11.8 K 2023-03-27 04:56:05,394 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~35.78 KB/36640, heapSize ~90.34 KB/92512, currentSize=84.81 KB/86846 for e7756c618a32a862ad613327bee74d79 in 280ms, sequenceid=146, compaction requested=true 2023-03-27 04:56:05,395 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:05,395 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-27 04:56:05,395 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-27 04:56:05,395 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-27 04:56:05,398 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,398 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:05,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,398 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:05,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:05,399 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:05,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:05,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:05,402 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,403 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=91.52 KB heapSize=258.28 KB 2023-03-27 04:56:05,406 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:05,406 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,406 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:05,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:05,407 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:05,407 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,407 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:05,407 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,407 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:05,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:05,407 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:05,407 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:05,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:05,407 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:05,407 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:05,407 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,408 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:05,407 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:05,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:05,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:05,408 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,408 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,408 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:05,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893025434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:05,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893025437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:05,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893025439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:05,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893025455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,461 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:05,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893025460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,479 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/ac325438900d406f81ba8d8dca300921 2023-03-27 04:56:05,510 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:05,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:05,515 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=32.34 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/646aa30c434441589179b9d3f3c68535 2023-03-27 04:56:05,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:05,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:05,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:05,518 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:05,518 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:05,518 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:05,518 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:05,519 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,519 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,520 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,520 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:05,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:05,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:05,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,520 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,520 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:05,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:05,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:05,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:05,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:05,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:05,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:05,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:05,522 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:05,522 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:05,522 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:05,522 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:05,522 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:05,523 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:05,523 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:05,523 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:05,523 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,523 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:05,523 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:05,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,524 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:05,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:05,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:05,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:05,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18182c01[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:05,529 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:05,529 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,537 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,537 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:05,537 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:05,537 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:05,538 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,538 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:05,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=32.34 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/5659f41a17d04a3a8a8e57f001d87d3d 2023-03-27 04:56:05,540 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:05,540 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,540 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:05,540 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:05,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:05,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:05,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893025542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:05,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:05,548 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:05,548 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:05,548 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:05,548 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:05,548 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:05,548 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/ac325438900d406f81ba8d8dca300921 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/ac325438900d406f81ba8d8dca300921 2023-03-27 04:56:05,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:05,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893025547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,553 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,553 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:05,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,554 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:05,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:05,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:05,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:05,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:05,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:05,556 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893025553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,560 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,563 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/cb431bf289ce4262ae1bd815e6d013d5 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/cb431bf289ce4262ae1bd815e6d013d5 2023-03-27 04:56:05,565 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/f3adbfa199c549f4a2b893331fb554a9 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/f3adbfa199c549f4a2b893331fb554a9 2023-03-27 04:56:05,565 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:05,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:05,567 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,567 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,568 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,568 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,568 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,568 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,568 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,567 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:05,568 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:05,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:05,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:05,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:05,568 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:05,568 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:05,568 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/ac325438900d406f81ba8d8dca300921, entries=150, sequenceid=212, filesize=11.8 K 2023-03-27 04:56:05,571 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/646aa30c434441589179b9d3f3c68535 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/646aa30c434441589179b9d3f3c68535 2023-03-27 04:56:05,581 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/B of e7756c618a32a862ad613327bee74d79 into cb431bf289ce4262ae1bd815e6d013d5(size=11.8 K), total size for store is 23.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:05,582 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:05,582 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/B, priority=13, startTime=1679892965057; duration=0sec 2023-03-27 04:56:05,582 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-27 04:56:05,582 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/646aa30c434441589179b9d3f3c68535, entries=150, sequenceid=212, filesize=11.8 K 2023-03-27 04:56:05,582 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-03-27 04:56:05,584 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/5659f41a17d04a3a8a8e57f001d87d3d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5659f41a17d04a3a8a8e57f001d87d3d 2023-03-27 04:56:05,587 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:05,588 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:05,588 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. because compaction request was cancelled 2023-03-27 04:56:05,588 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:56:05,590 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/A of e7756c618a32a862ad613327bee74d79 into f3adbfa199c549f4a2b893331fb554a9(size=11.8 K), total size for store is 37.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:05,590 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47832 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:56:05,590 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:05,590 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/C is initiating minor compaction (all files) 2023-03-27 04:56:05,590 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/A, priority=13, startTime=1679892965057; duration=0sec 2023-03-27 04:56:05,590 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/C in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:05,591 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/a107ec6aeaf64992adb1e32a594c0a44, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/4665b9efd869472498f9c6a7dc2ad8c2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/e01a3ce1cb584df3898c04eb8a3e8735, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/16ded63a4ec746d985aa1ad4f2094bfb] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=46.7 K 2023-03-27 04:56:05,591 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-27 04:56:05,591 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 4 compacting, 0 eligible, 16 blocking 2023-03-27 04:56:05,591 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:05,591 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:05,591 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. because compaction request was cancelled 2023-03-27 04:56:05,591 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:05,591 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting a107ec6aeaf64992adb1e32a594c0a44, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1679892961501 2023-03-27 04:56:05,592 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:05,596 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/B is initiating minor compaction (all files) 2023-03-27 04:56:05,597 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/B in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:05,597 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/cb431bf289ce4262ae1bd815e6d013d5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/9aa15ebb97f044dab136a04fb6eab2e1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/646aa30c434441589179b9d3f3c68535] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=35.3 K 2023-03-27 04:56:05,604 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting cb431bf289ce4262ae1bd815e6d013d5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1679892964225 2023-03-27 04:56:05,604 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 4665b9efd869472498f9c6a7dc2ad8c2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1679892963189 2023-03-27 04:56:05,607 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting e01a3ce1cb584df3898c04eb8a3e8735, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1679892964225 2023-03-27 04:56:05,607 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 9aa15ebb97f044dab136a04fb6eab2e1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1679892964972 2023-03-27 04:56:05,610 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 16ded63a4ec746d985aa1ad4f2094bfb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1679892964972 2023-03-27 04:56:05,611 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 646aa30c434441589179b9d3f3c68535, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1679892965389 2023-03-27 04:56:05,624 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5659f41a17d04a3a8a8e57f001d87d3d, entries=150, sequenceid=212, filesize=11.8 K 2023-03-27 04:56:05,626 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~98.23 KB/100586, heapSize ~278.16 KB/284832, currentSize=80.51 KB/82440 for e7756c618a32a862ad613327bee74d79 in 224ms, sequenceid=212, compaction requested=true 2023-03-27 04:56:05,627 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:05,627 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:05,627 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-27 04:56:05,627 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-27 04:56:05,634 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#B#compaction#30 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:05,634 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#C#compaction#29 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:05,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:05,663 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-27 04:56:05,664 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:05,664 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:05,664 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:05,664 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:05,664 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:05,664 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:05,668 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:05,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:05,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:05,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:05,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:05,679 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:05,679 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:05,679 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:05,679 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:05,679 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/93994c938ae946bcaa436935e0285573 2023-03-27 04:56:05,679 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,679 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,680 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,682 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:05,682 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,682 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:05,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:05,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:05,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:05,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:05,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:05,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:05,687 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:05,687 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:05,687 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:05,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:05,694 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:05,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:05,694 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:05,694 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:05,694 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:05,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:05,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:05,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,697 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:05,697 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:05,699 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,699 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:05,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:05,700 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,700 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:05,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:05,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:05,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:05,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:05,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b44613d[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:05,702 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:05,702 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,703 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,703 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:05,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:05,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:05,704 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,704 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:05,705 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/633f445a5f6a4641ad95ecdae1bacb11 2023-03-27 04:56:05,705 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:05,706 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:05,706 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:05,706 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:05,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:05,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:05,712 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:05,712 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:05,713 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:05,713 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:05,713 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:05,715 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,715 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,715 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:05,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:05,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,716 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:05,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,718 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:05,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:05,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,728 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:05,728 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:05,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:05,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:05,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:05,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:05,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:05,728 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:05,728 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,729 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:05,729 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:05,729 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:05,729 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:05,729 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,729 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,729 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:05,729 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,742 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/aabf3372470941ba9cd3ae716878338d 2023-03-27 04:56:05,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:05,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893025743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:05,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893025745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:05,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893025747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:05,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893025753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,756 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/93994c938ae946bcaa436935e0285573 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/93994c938ae946bcaa436935e0285573 2023-03-27 04:56:05,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:05,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893025760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,766 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/93994c938ae946bcaa436935e0285573, entries=200, sequenceid=230, filesize=14.1 K 2023-03-27 04:56:05,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/633f445a5f6a4641ad95ecdae1bacb11 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/633f445a5f6a4641ad95ecdae1bacb11 2023-03-27 04:56:05,778 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/633f445a5f6a4641ad95ecdae1bacb11, entries=150, sequenceid=230, filesize=11.8 K 2023-03-27 04:56:05,779 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/aabf3372470941ba9cd3ae716878338d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/aabf3372470941ba9cd3ae716878338d 2023-03-27 04:56:05,797 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/aabf3372470941ba9cd3ae716878338d, entries=150, sequenceid=230, filesize=11.8 K 2023-03-27 04:56:05,801 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for e7756c618a32a862ad613327bee74d79 in 138ms, sequenceid=230, compaction requested=true 2023-03-27 04:56:05,801 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:05,802 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-27 04:56:05,802 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=5), splitQueue=0 2023-03-27 04:56:05,802 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=6), splitQueue=0 2023-03-27 04:56:05,829 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:05,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:05,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:05,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:05,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:05,834 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:05,834 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:05,834 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:05,834 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:05,835 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,835 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,837 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:05,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:05,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:05,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,839 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:05,839 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,839 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:05,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:05,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:05,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:05,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:05,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:05,842 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:05,842 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:05,842 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:05,842 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:05,842 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:05,842 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:05,842 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:05,843 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-27 04:56:05,844 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:05,844 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:05,844 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:05,844 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:05,844 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:05,844 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:05,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:05,851 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:05,876 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/1d5e7284f91047c4a9ef37140c7031d9 2023-03-27 04:56:05,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:05,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893025886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:05,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:05,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893025888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:06,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:06,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893025998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893025998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:06,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893026056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:06,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893026059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,068 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/a4b82fe557e344ddab2f55a6257f97ef as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/a4b82fe557e344ddab2f55a6257f97ef 2023-03-27 04:56:06,070 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/698b505a609d4bbda96ace1a22f880c2 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/698b505a609d4bbda96ace1a22f880c2 2023-03-27 04:56:06,072 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:06,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893026067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,079 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/B of e7756c618a32a862ad613327bee74d79 into a4b82fe557e344ddab2f55a6257f97ef(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:06,082 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:06,083 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/B, priority=13, startTime=1679892965395; duration=0sec 2023-03-27 04:56:06,083 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=6), splitQueue=0 2023-03-27 04:56:06,083 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 4 compacting, 2 eligible, 16 blocking 2023-03-27 04:56:06,089 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:06,089 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:06,089 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. because compaction request was cancelled 2023-03-27 04:56:06,089 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 4 compacting, 2 eligible, 16 blocking 2023-03-27 04:56:06,093 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:06,093 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:06,093 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. because compaction request was cancelled 2023-03-27 04:56:06,093 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:56:06,093 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e7756c618a32a862ad613327bee74d79/C of e7756c618a32a862ad613327bee74d79 into 698b505a609d4bbda96ace1a22f880c2(size=12.0 K), total size for store is 35.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:06,094 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:06,094 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/C, priority=12, startTime=1679892965395; duration=0sec 2023-03-27 04:56:06,094 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-27 04:56:06,096 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53014 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:56:06,096 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/A is initiating minor compaction (all files) 2023-03-27 04:56:06,096 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 4 compacting, 0 eligible, 16 blocking 2023-03-27 04:56:06,096 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/A in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:06,096 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:06,096 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:06,096 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. because compaction request was cancelled 2023-03-27 04:56:06,096 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-27 04:56:06,096 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/f3adbfa199c549f4a2b893331fb554a9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d914c443343745ab8b2f769120fe59b0, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/ac325438900d406f81ba8d8dca300921, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/93994c938ae946bcaa436935e0285573] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=51.8 K 2023-03-27 04:56:06,097 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting f3adbfa199c549f4a2b893331fb554a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1679892964225 2023-03-27 04:56:06,098 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting d914c443343745ab8b2f769120fe59b0, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1679892964972 2023-03-27 04:56:06,098 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:06,099 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:06,099 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. because compaction request was cancelled 2023-03-27 04:56:06,099 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-27 04:56:06,099 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting ac325438900d406f81ba8d8dca300921, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1679892965389 2023-03-27 04:56:06,102 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:06,102 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 93994c938ae946bcaa436935e0285573, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679892965426 2023-03-27 04:56:06,102 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:06,103 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. because compaction request was cancelled 2023-03-27 04:56:06,114 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#A#compaction#36 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:06,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:06,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893026203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:06,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893026203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,320 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/8903952b934d4d5d89fe6a8151d60b02 2023-03-27 04:56:06,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:06,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893026508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:06,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893026508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,529 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/7b7d682992424da5a445310459d5ae50 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/7b7d682992424da5a445310459d5ae50 2023-03-27 04:56:06,538 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e7756c618a32a862ad613327bee74d79/A of e7756c618a32a862ad613327bee74d79 into 7b7d682992424da5a445310459d5ae50(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:06,538 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:06,538 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/A, priority=12, startTime=1679892965802; duration=0sec 2023-03-27 04:56:06,539 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:06,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:06,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893026566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:06,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893026568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:06,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893026579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,738 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/02214ebea5bd43a3a88a0559360a3f61 2023-03-27 04:56:06,746 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/1d5e7284f91047c4a9ef37140c7031d9 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/1d5e7284f91047c4a9ef37140c7031d9 2023-03-27 04:56:06,753 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/1d5e7284f91047c4a9ef37140c7031d9, entries=150, sequenceid=247, filesize=11.8 K 2023-03-27 04:56:06,757 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/8903952b934d4d5d89fe6a8151d60b02 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/8903952b934d4d5d89fe6a8151d60b02 2023-03-27 04:56:06,763 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/8903952b934d4d5d89fe6a8151d60b02, entries=150, sequenceid=247, filesize=11.8 K 2023-03-27 04:56:06,765 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/02214ebea5bd43a3a88a0559360a3f61 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/02214ebea5bd43a3a88a0559360a3f61 2023-03-27 04:56:06,776 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/02214ebea5bd43a3a88a0559360a3f61, entries=150, sequenceid=247, filesize=11.8 K 2023-03-27 04:56:06,777 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for e7756c618a32a862ad613327bee74d79 in 934ms, sequenceid=247, compaction requested=true 2023-03-27 04:56:06,777 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:06,777 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:06,777 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:06,778 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:06,778 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,779 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:06,779 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:06,780 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,780 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:06,780 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:06,780 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:06,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:06,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:06,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:06,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3af008fe[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:06,781 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:06,781 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:06,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:06,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:06,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:06,782 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:06,782 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:06,782 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,782 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:06,782 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:06,784 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,784 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:06,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:06,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:06,784 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:06,785 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:06,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:06,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:06,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:06,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:06,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:06,786 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:06,787 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:06,787 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:06,787 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:06,787 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:06,788 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:06,788 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:06,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:06,788 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:06,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:06,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:06,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:06,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:06,789 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:06,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:06,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,791 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:06,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,793 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,793 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:06,793 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:06,793 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,793 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:06,793 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:06,793 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:06,793 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:06,793 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:06,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:06,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:06,793 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:06,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:06,794 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:06,794 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:06,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:06,894 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:06,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:06,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:06,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:06,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:06,899 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:06,899 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:06,899 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:06,899 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:06,899 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:06,899 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:06,900 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:06,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:06,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:06,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:06,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:06,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:06,917 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:06,917 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:06,917 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:06,917 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:06,917 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:06,920 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:06,920 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:06,920 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:06,920 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:06,920 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-27 04:56:06,920 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:06,920 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:06,920 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:06,920 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:06,920 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:06,920 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:06,935 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/cbd9112850234648ad56b305389b186f 2023-03-27 04:56:06,963 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/bacbb614450b4b948b490b67950804e8 2023-03-27 04:56:07,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:07,015 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:07,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:07,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893027039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:07,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893027039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:07,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893027144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:07,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893027144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:07,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893027348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:07,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893027348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,390 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/48424308f6984b7eb29226b4a008d20c 2023-03-27 04:56:07,396 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/cbd9112850234648ad56b305389b186f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/cbd9112850234648ad56b305389b186f 2023-03-27 04:56:07,407 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/cbd9112850234648ad56b305389b186f, entries=150, sequenceid=265, filesize=11.9 K 2023-03-27 04:56:07,411 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/bacbb614450b4b948b490b67950804e8 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/bacbb614450b4b948b490b67950804e8 2023-03-27 04:56:07,429 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/bacbb614450b4b948b490b67950804e8, entries=150, sequenceid=265, filesize=11.9 K 2023-03-27 04:56:07,435 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/48424308f6984b7eb29226b4a008d20c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/48424308f6984b7eb29226b4a008d20c 2023-03-27 04:56:07,445 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/48424308f6984b7eb29226b4a008d20c, entries=150, sequenceid=265, filesize=11.9 K 2023-03-27 04:56:07,451 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for e7756c618a32a862ad613327bee74d79 in 531ms, sequenceid=265, compaction requested=true 2023-03-27 04:56:07,452 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:07,452 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:07,452 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:07,453 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:07,453 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,453 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:07,453 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:07,454 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,454 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:07,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:07,455 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,455 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:07,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:07,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:07,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:07,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:07,456 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:07,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e71fd29[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:07,456 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,457 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,457 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:07,457 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:07,457 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:07,458 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,458 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:07,459 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:07,459 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:07,459 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:07,460 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:07,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:07,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:07,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:07,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:07,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:07,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:07,463 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:07,463 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:07,463 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:07,463 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:07,463 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:07,464 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,465 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:07,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,465 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:07,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:07,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:07,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:07,466 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:07,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,469 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:07,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,478 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:07,478 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:07,478 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:07,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:07,478 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:07,478 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:07,478 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,478 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,478 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:07,478 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:07,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:07,478 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:07,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:07,479 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:07,479 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:07,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:07,579 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-27 04:56:07,579 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:07,579 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:07,579 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:07,579 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:07,580 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:07,580 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:07,580 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:07,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:07,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:07,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:07,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:07,585 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:07,585 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:07,585 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:07,585 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:07,586 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,586 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:07,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:07,587 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:07,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:07,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:07,587 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,587 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:07,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:07,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:07,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:07,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:07,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893027602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:07,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:07,606 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:07,606 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:07,606 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:07,610 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:07,610 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:07,610 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:07,610 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:07,610 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:07,610 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:07,610 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:07,610 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:07,610 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,610 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:07,610 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:07,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:07,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:07,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893027604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893027604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,616 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,616 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:07,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:07,617 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:07,617 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:07,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:07,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:07,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:07,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57d0a0c2[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:07,618 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:07,618 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,619 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,620 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:07,620 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:07,620 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:07,624 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,624 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:07,625 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/2563e834e7394560b8cf8cc3529e9721 2023-03-27 04:56:07,628 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:07,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:07,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:07,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:07,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:07,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:07,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:07,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:07,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:07,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:07,631 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:07,631 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:07,631 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:07,631 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:07,631 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:07,640 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,640 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:07,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:07,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:07,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,641 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:07,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:07,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:07,644 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:07,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,646 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:07,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,649 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,649 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:07,649 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:07,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:07,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:07,650 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:07,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,652 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:07,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:07,652 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:07,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:07,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:07,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:07,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:07,654 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:07,654 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:07,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893027653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:07,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893027654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,660 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/b16cf732665f44c8a1ebb36df7d5f845 2023-03-27 04:56:07,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:07,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893027705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:07,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893027713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:07,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893027715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,754 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:07,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:07,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:07,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:07,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:07,758 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:07,758 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:07,758 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:07,758 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:07,759 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,759 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:07,759 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:07,759 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:07,760 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,760 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:07,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:07,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:07,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:07,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:07,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:07,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:07,761 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:07,761 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:07,761 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:07,761 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:07,761 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:07,761 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:07,761 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:07,761 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:07,761 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:07,761 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:07,761 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:07,761 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,761 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:07,761 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:07,763 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,763 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:07,764 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,764 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:07,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:07,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ff57e8c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:07,765 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:07,765 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,766 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,766 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:07,766 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:07,766 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,766 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:07,766 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:07,767 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,767 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:07,767 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:07,767 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:07,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:07,769 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:07,769 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:07,769 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:07,769 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:07,769 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:07,770 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,770 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:07,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:07,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,770 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:07,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:07,770 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:07,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:07,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:07,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,771 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:07,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,773 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,773 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:07,773 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:07,774 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:07,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:07,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:07,774 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:07,773 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:07,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:07,774 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,774 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,774 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,774 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,774 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:07,774 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:07,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:07,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:07,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:07,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:07,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:07,874 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:07,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:07,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:07,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:07,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:07,879 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:07,879 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:07,879 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:07,879 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:07,879 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,879 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:07,880 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:07,880 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:07,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:07,881 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,881 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:07,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:07,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:07,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:07,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:07,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:07,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:07,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:07,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:07,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:07,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:07,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:07,882 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:07,882 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:07,882 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:07,882 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:07,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:07,883 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:07,883 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,883 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:07,883 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:07,884 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,884 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:07,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:07,886 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,886 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:07,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:07,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:07,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:07,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:07,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f82674[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:07,887 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:07,887 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,888 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,888 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:07,888 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:07,888 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:07,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:07,889 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,889 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:07,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:07,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:07,889 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:07,890 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:07,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:07,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:07,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:07,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:07,892 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:07,892 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:07,892 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:07,892 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:07,892 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:07,893 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,893 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:07,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,894 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:07,894 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:07,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:07,896 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:07,911 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:07,911 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:07,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:07,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:07,912 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,912 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:07,912 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:07,912 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:07,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:07,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:07,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:07,912 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:07,912 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:07,912 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:07,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:07,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893027912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:07,912 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,913 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:07,913 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:07,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:07,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893027920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:07,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:07,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893027920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,013 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:08,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:08,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:08,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:08,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:08,016 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:08,016 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:08,016 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:08,016 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:08,017 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,017 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:08,018 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:08,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:08,018 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,018 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:08,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:08,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:08,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:08,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:08,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:08,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:08,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:08,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:08,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:08,019 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:08,019 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:08,019 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:08,019 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:08,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:08,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:08,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:08,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:08,020 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,020 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:08,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:08,021 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,021 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:08,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:08,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e10b57[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:08,022 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:08,022 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,025 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,025 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:08,025 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:08,025 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,025 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:08,025 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:08,026 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:08,026 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,026 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:08,026 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:08,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:08,029 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:08,029 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:08,029 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:08,029 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:08,029 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:08,030 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:08,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:08,030 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,030 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:08,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,031 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,032 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:08,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,035 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:08,035 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:08,035 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:08,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:08,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:08,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:08,035 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:08,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:08,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:08,035 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,036 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:08,036 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,036 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,036 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:08,036 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,036 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,036 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,095 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/bdc3f5acfe394173b0b919de7daa4003 2023-03-27 04:56:08,101 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/2563e834e7394560b8cf8cc3529e9721 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/2563e834e7394560b8cf8cc3529e9721 2023-03-27 04:56:08,107 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/2563e834e7394560b8cf8cc3529e9721, entries=150, sequenceid=283, filesize=11.9 K 2023-03-27 04:56:08,108 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/b16cf732665f44c8a1ebb36df7d5f845 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/b16cf732665f44c8a1ebb36df7d5f845 2023-03-27 04:56:08,115 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/b16cf732665f44c8a1ebb36df7d5f845, entries=150, sequenceid=283, filesize=11.9 K 2023-03-27 04:56:08,116 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/bdc3f5acfe394173b0b919de7daa4003 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bdc3f5acfe394173b0b919de7daa4003 2023-03-27 04:56:08,122 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bdc3f5acfe394173b0b919de7daa4003, entries=150, sequenceid=283, filesize=11.9 K 2023-03-27 04:56:08,123 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for e7756c618a32a862ad613327bee74d79 in 544ms, sequenceid=283, compaction requested=true 2023-03-27 04:56:08,123 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:08,123 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:08,123 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:56:08,123 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-27 04:56:08,123 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:08,124 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:08,125 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48820 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:56:08,125 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/A is initiating minor compaction (all files) 2023-03-27 04:56:08,125 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/A in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:08,126 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/7b7d682992424da5a445310459d5ae50, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/1d5e7284f91047c4a9ef37140c7031d9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/cbd9112850234648ad56b305389b186f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/2563e834e7394560b8cf8cc3529e9721] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=47.7 K 2023-03-27 04:56:08,126 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60822 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-27 04:56:08,126 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/B is initiating minor compaction (all files) 2023-03-27 04:56:08,126 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/B in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:08,126 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/a4b82fe557e344ddab2f55a6257f97ef, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/633f445a5f6a4641ad95ecdae1bacb11, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/8903952b934d4d5d89fe6a8151d60b02, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/bacbb614450b4b948b490b67950804e8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/b16cf732665f44c8a1ebb36df7d5f845] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=59.4 K 2023-03-27 04:56:08,126 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 7b7d682992424da5a445310459d5ae50, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679892965434 2023-03-27 04:56:08,128 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting a4b82fe557e344ddab2f55a6257f97ef, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1679892965389 2023-03-27 04:56:08,129 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 1d5e7284f91047c4a9ef37140c7031d9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679892965730 2023-03-27 04:56:08,129 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 633f445a5f6a4641ad95ecdae1bacb11, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679892965434 2023-03-27 04:56:08,129 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting cbd9112850234648ad56b305389b186f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1679892965874 2023-03-27 04:56:08,130 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 8903952b934d4d5d89fe6a8151d60b02, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679892965730 2023-03-27 04:56:08,130 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 2563e834e7394560b8cf8cc3529e9721, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679892967021 2023-03-27 04:56:08,130 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting bacbb614450b4b948b490b67950804e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1679892965874 2023-03-27 04:56:08,131 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting b16cf732665f44c8a1ebb36df7d5f845, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679892967021 2023-03-27 04:56:08,136 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:08,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:08,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:08,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:08,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:08,141 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:08,141 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:08,141 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:08,141 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:08,142 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,142 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,144 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:08,144 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:08,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,145 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,145 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:08,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:08,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:08,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:08,150 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:08,150 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:08,150 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:08,153 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#B#compaction#44 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:08,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:08,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:08,158 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:08,158 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:08,158 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#A#compaction#45 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:08,158 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-27 04:56:08,159 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:08,159 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:08,159 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:08,159 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:08,159 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:08,159 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:08,168 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:08,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:08,197 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/25708bc12dea4ec58493998793728f16 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/25708bc12dea4ec58493998793728f16 2023-03-27 04:56:08,208 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e7756c618a32a862ad613327bee74d79/B of e7756c618a32a862ad613327bee74d79 into 25708bc12dea4ec58493998793728f16(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:08,208 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:08,208 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/B, priority=11, startTime=1679892968123; duration=0sec 2023-03-27 04:56:08,209 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:08,209 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-27 04:56:08,212 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72790 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-27 04:56:08,212 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/C is initiating minor compaction (all files) 2023-03-27 04:56:08,212 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/C in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:08,213 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/698b505a609d4bbda96ace1a22f880c2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5659f41a17d04a3a8a8e57f001d87d3d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/aabf3372470941ba9cd3ae716878338d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/02214ebea5bd43a3a88a0559360a3f61, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/48424308f6984b7eb29226b4a008d20c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bdc3f5acfe394173b0b919de7daa4003] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=71.1 K 2023-03-27 04:56:08,214 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 698b505a609d4bbda96ace1a22f880c2, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1679892964972 2023-03-27 04:56:08,215 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 5659f41a17d04a3a8a8e57f001d87d3d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1679892965389 2023-03-27 04:56:08,216 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting aabf3372470941ba9cd3ae716878338d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679892965434 2023-03-27 04:56:08,217 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 02214ebea5bd43a3a88a0559360a3f61, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679892965730 2023-03-27 04:56:08,218 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 48424308f6984b7eb29226b4a008d20c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1679892965874 2023-03-27 04:56:08,219 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting bdc3f5acfe394173b0b919de7daa4003, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679892967021 2023-03-27 04:56:08,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893028225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893028238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,253 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#C#compaction#47 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:08,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893028243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893028247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893028247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,267 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/5ea2f681a5824941919d3f3ad82e7693 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5ea2f681a5824941919d3f3ad82e7693 2023-03-27 04:56:08,274 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e7756c618a32a862ad613327bee74d79/C of e7756c618a32a862ad613327bee74d79 into 5ea2f681a5824941919d3f3ad82e7693(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:08,274 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:08,274 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/C, priority=10, startTime=1679892968124; duration=0sec 2023-03-27 04:56:08,275 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:08,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893028356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893028356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893028356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893028359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893028560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893028561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893028562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893028564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,594 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/c10aeccba94645e29670b453be1207e9 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c10aeccba94645e29670b453be1207e9 2023-03-27 04:56:08,613 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e7756c618a32a862ad613327bee74d79/A of e7756c618a32a862ad613327bee74d79 into c10aeccba94645e29670b453be1207e9(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:08,613 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:08,613 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/A, priority=12, startTime=1679892968123; duration=0sec 2023-03-27 04:56:08,613 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:08,621 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/73c035ebac834991a52dd29bc11502f7 2023-03-27 04:56:08,639 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/57a46d0439bf4c9aa3ca358aee49eb02 2023-03-27 04:56:08,669 WARN [DataStreamer for file /user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/3a91f7aabb094ae68185c2cfcb448f5d] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-03-27 04:56:08,670 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/3a91f7aabb094ae68185c2cfcb448f5d 2023-03-27 04:56:08,676 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/73c035ebac834991a52dd29bc11502f7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/73c035ebac834991a52dd29bc11502f7 2023-03-27 04:56:08,682 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/73c035ebac834991a52dd29bc11502f7, entries=150, sequenceid=297, filesize=11.9 K 2023-03-27 04:56:08,683 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/57a46d0439bf4c9aa3ca358aee49eb02 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/57a46d0439bf4c9aa3ca358aee49eb02 2023-03-27 04:56:08,689 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/57a46d0439bf4c9aa3ca358aee49eb02, entries=150, sequenceid=297, filesize=11.9 K 2023-03-27 04:56:08,698 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/3a91f7aabb094ae68185c2cfcb448f5d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/3a91f7aabb094ae68185c2cfcb448f5d 2023-03-27 04:56:08,705 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/3a91f7aabb094ae68185c2cfcb448f5d, entries=150, sequenceid=297, filesize=11.9 K 2023-03-27 04:56:08,710 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for e7756c618a32a862ad613327bee74d79 in 552ms, sequenceid=297, compaction requested=false 2023-03-27 04:56:08,711 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:08,711 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:08,711 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:08,711 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:08,711 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,711 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:08,711 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:08,719 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:08,720 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,720 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:08,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:08,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56657db9[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:08,721 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:08,721 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,722 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,722 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:08,722 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:08,722 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:08,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:08,726 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:08,726 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:08,726 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:08,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:08,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:08,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:08,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:08,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:08,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:08,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:08,729 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:08,729 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:08,729 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:08,729 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:08,729 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:08,745 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,745 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:08,745 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,746 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:08,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:08,748 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,756 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:08,757 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:08,757 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,757 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,757 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:08,757 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:08,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:08,757 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,757 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,757 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,757 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:08,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:08,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:08,757 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:08,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:08,758 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:08,761 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-27 04:56:08,761 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:08,761 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:08,761 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:08,761 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:08,761 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:08,762 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:08,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:08,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893028804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,858 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:08,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:08,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:08,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:08,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:08,861 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:08,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:08,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:08,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:08,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:08,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:08,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:08,865 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,866 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,866 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:08,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893028866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:08,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:08,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:08,867 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:08,867 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:08,867 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:08,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:08,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:08,868 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:08,868 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:08,868 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:08,868 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:08,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:08,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:08,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:08,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:08,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893028867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893028867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893028868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,871 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,871 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:08,871 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,871 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:08,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:08,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:08,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:08,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@427078e7[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:08,873 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:08,873 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,875 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:08,875 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:08,875 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:08,875 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,876 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:08,876 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:08,876 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:08,876 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,876 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:08,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:08,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:08,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:08,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:08,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:08,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:08,879 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:08,879 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:08,879 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:08,879 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:08,879 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:08,880 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,880 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:08,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:08,881 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:08,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:08,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:08,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:08,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,881 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:08,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,882 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:08,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:08,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,884 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,884 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,884 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,884 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:08,884 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,884 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:08,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:08,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:08,884 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:08,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:08,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:08,884 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,886 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:08,886 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:08,886 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:08,886 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:08,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:08,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:08,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:08,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893028907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,986 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:08,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:08,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:08,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:08,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:08,993 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:08,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:08,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:08,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:08,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,994 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:08,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:08,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:08,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:08,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:08,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:08,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:08,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:08,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:09,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:09,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:09,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:09,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:09,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:09,009 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:09,009 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:09,009 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:09,009 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:09,009 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:09,009 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:09,009 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:09,009 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:09,010 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,010 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:09,010 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:09,011 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,011 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,011 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,011 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:09,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67f2d5b[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:09,015 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:09,015 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,016 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,016 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:09,016 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,016 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:09,016 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:09,016 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:09,017 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:09,017 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:09,017 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:09,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:09,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:09,020 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:09,020 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:09,020 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:09,021 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:09,021 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:09,028 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,028 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,028 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:09,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:09,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:09,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,034 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,036 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,040 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:09,040 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,040 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:09,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:09,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:09,040 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,040 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,040 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:09,040 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,040 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,040 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,040 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:09,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:09,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:09,041 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:09,041 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:09,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:09,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:09,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:09,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893029111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,141 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:09,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:09,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:09,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:09,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:09,146 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:09,146 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:09,146 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:09,146 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:09,146 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,146 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,148 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:09,148 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:09,148 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,148 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:09,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:09,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:09,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:09,151 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:09,151 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:09,151 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:09,153 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:09,153 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:09,153 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:09,153 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:09,153 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:09,153 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:09,153 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:09,153 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:09,153 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,153 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:09,153 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:09,161 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,161 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,161 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:09,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:09,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a482570[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:09,165 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:09,165 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,178 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,179 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:09,179 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:09,179 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:09,179 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,179 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:09,181 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,181 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,181 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:09,182 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:09,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:09,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:09,184 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:09,184 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:09,184 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:09,184 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:09,184 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:09,186 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,186 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,186 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,187 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,190 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,199 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,199 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,199 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,199 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,199 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:09,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:09,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:09,199 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:09,199 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:09,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:09,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:09,199 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:09,199 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:09,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:09,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:09,206 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/d155a5e080fb47aa90ded1b62940711a 2023-03-27 04:56:09,242 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/34ab99085ceb4d74a9d04b1dc7d28862 2023-03-27 04:56:09,289 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/6ac5e06a05854070bfce4feba1b282a4 2023-03-27 04:56:09,299 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:09,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:09,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:09,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:09,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:09,306 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:09,306 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:09,306 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:09,306 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:09,306 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,306 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,307 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:09,307 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:09,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:09,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,307 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,308 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:09,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:09,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:09,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:09,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:09,309 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:09,309 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:09,309 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:09,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/d155a5e080fb47aa90ded1b62940711a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d155a5e080fb47aa90ded1b62940711a 2023-03-27 04:56:09,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:09,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:09,311 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:09,311 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:09,311 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:09,311 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:09,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:09,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:09,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,312 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:09,312 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:09,315 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,315 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,315 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,315 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,316 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d155a5e080fb47aa90ded1b62940711a, entries=200, sequenceid=319, filesize=14.3 K 2023-03-27 04:56:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,318 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/34ab99085ceb4d74a9d04b1dc7d28862 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/34ab99085ceb4d74a9d04b1dc7d28862 2023-03-27 04:56:09,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:09,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@773d0935[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:09,319 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:09,319 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,325 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/34ab99085ceb4d74a9d04b1dc7d28862, entries=150, sequenceid=319, filesize=11.9 K 2023-03-27 04:56:09,328 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/6ac5e06a05854070bfce4feba1b282a4 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/6ac5e06a05854070bfce4feba1b282a4 2023-03-27 04:56:09,336 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,336 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:09,337 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:09,337 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,337 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:09,337 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:09,337 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/6ac5e06a05854070bfce4feba1b282a4, entries=150, sequenceid=319, filesize=11.9 K 2023-03-27 04:56:09,338 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:09,338 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:09,338 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,338 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:09,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,339 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for e7756c618a32a862ad613327bee74d79 in 578ms, sequenceid=319, compaction requested=true 2023-03-27 04:56:09,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,339 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:09,340 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:09,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:09,340 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:09,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:09,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,341 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39510 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:09,342 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,342 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/B is initiating minor compaction (all files) 2023-03-27 04:56:09,342 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/A is initiating minor compaction (all files) 2023-03-27 04:56:09,342 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/B in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:09,342 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/A in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:09,342 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/25708bc12dea4ec58493998793728f16, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/57a46d0439bf4c9aa3ca358aee49eb02, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/34ab99085ceb4d74a9d04b1dc7d28862] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=36.2 K 2023-03-27 04:56:09,342 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c10aeccba94645e29670b453be1207e9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/73c035ebac834991a52dd29bc11502f7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d155a5e080fb47aa90ded1b62940711a] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=38.6 K 2023-03-27 04:56:09,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,343 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 25708bc12dea4ec58493998793728f16, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679892967021 2023-03-27 04:56:09,343 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting c10aeccba94645e29670b453be1207e9, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679892967021 2023-03-27 04:56:09,343 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 57a46d0439bf4c9aa3ca358aee49eb02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1679892967600 2023-03-27 04:56:09,343 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 73c035ebac834991a52dd29bc11502f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1679892967600 2023-03-27 04:56:09,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:09,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:09,344 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:09,344 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:09,344 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:09,344 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:09,344 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:09,347 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 34ab99085ceb4d74a9d04b1dc7d28862, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1679892968236 2023-03-27 04:56:09,347 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting d155a5e080fb47aa90ded1b62940711a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1679892968236 2023-03-27 04:56:09,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,359 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,360 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:09,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:09,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:09,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:09,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:09,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,365 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:09,365 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:09,365 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:09,366 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#B#compaction#53 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:09,367 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:09,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:09,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:09,375 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#A#compaction#54 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:09,375 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-27 04:56:09,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:09,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:09,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:09,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:09,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:09,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:09,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:09,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893029402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:09,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893029405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:09,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893029409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:09,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893029410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:09,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893029413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,428 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/99fee2c7e86e4607b465b6e67ff9d35c 2023-03-27 04:56:09,450 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/a845b5eea35748a08700dc3b71af75b9 2023-03-27 04:56:09,466 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:09,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:09,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:09,470 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/15ccc8f8855e4df3957e0f54f37b1154 2023-03-27 04:56:09,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:09,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:09,478 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:09,478 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:09,478 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:09,478 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:09,479 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,479 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,479 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:09,479 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:09,480 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,480 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:09,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:09,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:09,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:09,481 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:09,481 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:09,481 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:09,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/99fee2c7e86e4607b465b6e67ff9d35c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/99fee2c7e86e4607b465b6e67ff9d35c 2023-03-27 04:56:09,494 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:09,500 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:09,500 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:09,500 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:09,500 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:09,500 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:09,500 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:09,500 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:09,500 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,500 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:09,500 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:09,499 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/99fee2c7e86e4607b465b6e67ff9d35c, entries=150, sequenceid=334, filesize=11.9 K 2023-03-27 04:56:09,502 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,502 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,502 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/a845b5eea35748a08700dc3b71af75b9 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/a845b5eea35748a08700dc3b71af75b9 2023-03-27 04:56:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,503 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,503 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:09,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36cbf075[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:09,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:09,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,505 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,506 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:09,506 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:09,506 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,506 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:09,506 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:09,507 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,507 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:09,507 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:09,507 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:09,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:09,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:09,509 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:09,510 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:09,510 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:09,510 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:09,510 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:09,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:09,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:09,513 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,514 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,515 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/a845b5eea35748a08700dc3b71af75b9, entries=150, sequenceid=334, filesize=11.9 K 2023-03-27 04:56:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:09,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:09,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:09,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:09,517 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:09,517 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:09,517 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,517 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/15ccc8f8855e4df3957e0f54f37b1154 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/15ccc8f8855e4df3957e0f54f37b1154 2023-03-27 04:56:09,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,518 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:09,518 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:09,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:09,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:09,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:09,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893029512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,519 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:09,519 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,519 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,519 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:09,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:09,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893029512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:09,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893029518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/15ccc8f8855e4df3957e0f54f37b1154, entries=150, sequenceid=334, filesize=11.9 K 2023-03-27 04:56:09,529 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for e7756c618a32a862ad613327bee74d79 in 154ms, sequenceid=334, compaction requested=true 2023-03-27 04:56:09,529 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:09,529 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-27 04:56:09,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-27 04:56:09,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-27 04:56:09,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:09,531 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-27 04:56:09,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:09,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:09,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:09,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:09,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:09,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:09,567 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/b64a3a320207416ba8cc68af0323f875 2023-03-27 04:56:09,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:09,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49510 deadline: 1679893029608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,618 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:09,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:09,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:09,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:09,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:09,635 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:09,635 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:09,635 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:09,635 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:09,636 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,636 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,637 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:09,637 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:09,637 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,637 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:09,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:09,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:09,646 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:09,646 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:09,646 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:09,651 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:09,651 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:09,651 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:09,651 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:09,651 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:09,651 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:09,651 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:09,651 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:09,652 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,652 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:09,652 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:09,653 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,653 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,653 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,653 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:09,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:09,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49211602[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:09,655 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:09,655 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,656 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,656 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:09,656 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:09,656 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:09,657 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,657 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:09,658 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:09,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:09,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:09,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:09,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:09,661 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:09,661 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:09,661 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:09,661 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:09,661 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:09,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:09,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,666 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:09,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:09,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,668 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,679 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:09,679 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:09,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:09,679 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:09,679 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:09,679 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:09,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:09,680 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,680 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,680 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,680 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,680 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,680 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:09,680 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:09,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:09,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:09,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:09,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:09,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49510 deadline: 1679893029718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:09,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893029722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:09,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893029723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:09,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893029737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,780 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:09,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:09,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:09,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:09,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:09,799 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:09,799 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:09,800 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:09,800 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:09,800 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,800 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,801 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:09,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:09,801 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:09,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,802 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,802 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:09,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:09,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:09,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:09,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:09,806 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:09,806 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:09,806 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:09,809 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:09,809 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:09,809 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:09,809 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:09,809 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:09,810 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:09,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:09,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:09,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:09,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:09,822 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/0a8888ea92ef4150975bdf366ab492c5 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/0a8888ea92ef4150975bdf366ab492c5 2023-03-27 04:56:09,823 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,823 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,824 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,824 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:09,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@289c9f3[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:09,825 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:09,826 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,828 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,828 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:09,828 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:09,828 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:09,828 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,828 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:09,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,833 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:09,833 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:09,833 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:09,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,837 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/B of e7756c618a32a862ad613327bee74d79 into 0a8888ea92ef4150975bdf366ab492c5(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:09,837 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:09,837 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/B, priority=13, startTime=1679892969340; duration=0sec 2023-03-27 04:56:09,838 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-27 04:56:09,838 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 3 compacting, 1 eligible, 16 blocking 2023-03-27 04:56:09,839 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:09,839 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:09,839 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. because compaction request was cancelled 2023-03-27 04:56:09,839 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:56:09,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:09,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:09,840 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:09,840 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:09,840 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:09,840 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:09,840 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:09,841 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/9cdd728b094e42d781c7c0186ad26de2 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/9cdd728b094e42d781c7c0186ad26de2 2023-03-27 04:56:09,841 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:56:09,842 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/C is initiating minor compaction (all files) 2023-03-27 04:56:09,842 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/C in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:09,842 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5ea2f681a5824941919d3f3ad82e7693, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/3a91f7aabb094ae68185c2cfcb448f5d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/6ac5e06a05854070bfce4feba1b282a4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/15ccc8f8855e4df3957e0f54f37b1154] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=48.1 K 2023-03-27 04:56:09,842 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 5ea2f681a5824941919d3f3ad82e7693, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679892967021 2023-03-27 04:56:09,843 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 3a91f7aabb094ae68185c2cfcb448f5d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1679892967600 2023-03-27 04:56:09,844 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 6ac5e06a05854070bfce4feba1b282a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1679892968236 2023-03-27 04:56:09,844 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 15ccc8f8855e4df3957e0f54f37b1154, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679892968787 2023-03-27 04:56:09,845 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,845 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,845 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:09,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,846 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,850 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,853 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/A of e7756c618a32a862ad613327bee74d79 into 9cdd728b094e42d781c7c0186ad26de2(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:09,853 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:09,853 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/A, priority=13, startTime=1679892969339; duration=0sec 2023-03-27 04:56:09,853 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-27 04:56:09,853 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 4 compacting, 0 eligible, 16 blocking 2023-03-27 04:56:09,853 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:09,853 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:09,853 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. because compaction request was cancelled 2023-03-27 04:56:09,853 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-27 04:56:09,854 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:09,854 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:09,855 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. because compaction request was cancelled 2023-03-27 04:56:09,856 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,856 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,856 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,856 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,856 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:09,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:09,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:09,856 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,857 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:09,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:09,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:09,857 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:09,857 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:09,857 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:09,857 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:09,857 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:09,866 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#C#compaction#60 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:09,904 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/fea5d17b9cf84edbad27556e8bee3434 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/fea5d17b9cf84edbad27556e8bee3434 2023-03-27 04:56:09,917 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e7756c618a32a862ad613327bee74d79/C of e7756c618a32a862ad613327bee74d79 into fea5d17b9cf84edbad27556e8bee3434(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:09,917 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:09,918 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/C, priority=12, startTime=1679892969530; duration=0sec 2023-03-27 04:56:09,918 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:09,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:09,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893029919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:09,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49510 deadline: 1679893029924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,957 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:09,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:09,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:09,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:09,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:09,966 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:09,966 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:09,966 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:09,966 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:09,967 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,967 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,967 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:09,967 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:09,968 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,968 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:09,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:09,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:09,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:09,970 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:09,970 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:09,970 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:09,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:09,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:09,971 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:09,972 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:09,972 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:09,972 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:09,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:09,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:09,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:09,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:09,973 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,973 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,974 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,974 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:09,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:09,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d621c36[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:09,975 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:09,975 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,976 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:09,977 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:09,977 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:09,977 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:09,977 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,977 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:09,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,978 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:09,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,978 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:09,978 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:09,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:09,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:09,990 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:09,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:09,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:09,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:09,990 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:09,993 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/e3cf67a406904dcdb5fb18419d67df62 2023-03-27 04:56:09,995 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:09,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:09,996 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,996 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,996 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:09,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:09,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:09,998 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:09,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:09,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,011 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:10,011 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:10,011 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:10,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:10,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:10,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:10,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:10,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:10,012 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:10,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:10,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:10,012 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:10,012 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,012 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,012 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,012 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,012 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,012 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,012 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:10,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49536 deadline: 1679893030025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:10,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893030030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:10,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893030040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,112 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:10,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:10,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:10,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:10,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:10,116 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:10,116 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:10,116 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:10,116 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:10,116 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,116 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:10,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:10,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:10,117 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,118 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,118 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:10,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:10,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:10,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:10,119 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:10,119 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:10,119 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:10,119 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:10,119 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:10,119 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:10,120 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:10,120 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:10,120 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:10,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:10,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:10,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:10,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:10,131 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,131 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,132 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,132 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:10,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:10,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74431fb7[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:10,133 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:10,133 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,134 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,134 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:10,134 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:10,134 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:10,136 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,136 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:10,138 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:10,138 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:10,138 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,138 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:10,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:10,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:10,142 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:10,142 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:10,142 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:10,142 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:10,142 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:10,148 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,148 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,148 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:10,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:10,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:10,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,150 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,152 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,157 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,157 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,157 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:10,157 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:10,157 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,157 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:10,157 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:10,157 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:10,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:10,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:10,157 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,158 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,158 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:10,158 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:10,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:10,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:10,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:10,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:10,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49510 deadline: 1679893030228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,258 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:10,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:10,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:10,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:10,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:10,261 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:10,262 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:10,262 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:10,262 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:10,262 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,262 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,263 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:10,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:10,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:10,263 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,263 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,263 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:10,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:10,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:10,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:10,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:10,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:10,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:10,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:10,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:10,264 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:10,264 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:10,264 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:10,264 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:10,265 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:10,265 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:10,265 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,265 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:10,265 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:10,265 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,265 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,266 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,266 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:10,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:10,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3513963b[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:10,273 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:10,273 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,290 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,290 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:10,290 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:10,290 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:10,290 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:10,295 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,295 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,295 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,295 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:10,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:10,298 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:10,298 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:10,298 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:10,298 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:10,298 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:10,299 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,299 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:10,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,299 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:10,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:10,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,301 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,311 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,313 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:10,313 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:10,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:10,313 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:10,313 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:10,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:10,313 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,313 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:10,313 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,313 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,313 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,313 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:10,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:10,313 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:10,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:10,313 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:10,413 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:10,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:10,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:10,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:10,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:10,422 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:10,422 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:10,422 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:10,422 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:10,422 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,423 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,423 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,423 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:10,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:10,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:10,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,424 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,424 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:10,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:10,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:10,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:10,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:10,426 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:10,426 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:10,426 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:10,426 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/4ff6928b52a74108943ca02eed9dca48 2023-03-27 04:56:10,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:10,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:10,427 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:10,427 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:10,427 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:10,427 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:10,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:10,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:10,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:10,428 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:10,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,429 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:10,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,434 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/b64a3a320207416ba8cc68af0323f875 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/b64a3a320207416ba8cc68af0323f875 2023-03-27 04:56:10,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:10,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7999073b[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:10,435 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:10,436 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,437 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,438 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:10,438 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:10,438 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:10,439 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,439 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:10,439 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,440 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:10,440 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:10,440 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:10,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,444 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/b64a3a320207416ba8cc68af0323f875, entries=200, sequenceid=352, filesize=14.3 K 2023-03-27 04:56:10,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,446 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/e3cf67a406904dcdb5fb18419d67df62 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/e3cf67a406904dcdb5fb18419d67df62 2023-03-27 04:56:10,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:10,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:10,447 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:10,447 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:10,447 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:10,447 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:10,447 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:10,456 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,456 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,456 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,457 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/e3cf67a406904dcdb5fb18419d67df62, entries=150, sequenceid=352, filesize=11.9 K 2023-03-27 04:56:10,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:10,457 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,459 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/4ff6928b52a74108943ca02eed9dca48 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/4ff6928b52a74108943ca02eed9dca48 2023-03-27 04:56:10,459 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:10,461 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:10,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:10,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:10,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:10,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:10,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:10,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,462 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,462 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:10,462 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:10,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:10,466 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/4ff6928b52a74108943ca02eed9dca48, entries=150, sequenceid=352, filesize=11.9 K 2023-03-27 04:56:10,466 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for e7756c618a32a862ad613327bee74d79 in 935ms, sequenceid=352, compaction requested=true 2023-03-27 04:56:10,467 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:10,467 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:10,467 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:10,467 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:10,467 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:10,467 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:10,468 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39612 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:10,468 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:10,468 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/A is initiating minor compaction (all files) 2023-03-27 04:56:10,468 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/B is initiating minor compaction (all files) 2023-03-27 04:56:10,468 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/A in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:10,468 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/B in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:10,468 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/9cdd728b094e42d781c7c0186ad26de2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/99fee2c7e86e4607b465b6e67ff9d35c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/b64a3a320207416ba8cc68af0323f875] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=38.7 K 2023-03-27 04:56:10,468 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/0a8888ea92ef4150975bdf366ab492c5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/a845b5eea35748a08700dc3b71af75b9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/e3cf67a406904dcdb5fb18419d67df62] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=36.3 K 2023-03-27 04:56:10,469 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 9cdd728b094e42d781c7c0186ad26de2, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1679892968236 2023-03-27 04:56:10,469 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 0a8888ea92ef4150975bdf366ab492c5, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1679892968236 2023-03-27 04:56:10,469 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 99fee2c7e86e4607b465b6e67ff9d35c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679892968787 2023-03-27 04:56:10,469 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting a845b5eea35748a08700dc3b71af75b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679892968787 2023-03-27 04:56:10,469 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting b64a3a320207416ba8cc68af0323f875, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1679892969400 2023-03-27 04:56:10,470 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting e3cf67a406904dcdb5fb18419d67df62, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1679892969400 2023-03-27 04:56:10,483 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#B#compaction#62 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:10,495 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#A#compaction#63 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:10,519 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/fc1ef4bb68e1428a943a43f06fb879b8 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/fc1ef4bb68e1428a943a43f06fb879b8 2023-03-27 04:56:10,522 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/97298b673b1a497ca3c72f6746494f5d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/97298b673b1a497ca3c72f6746494f5d 2023-03-27 04:56:10,530 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/B of e7756c618a32a862ad613327bee74d79 into fc1ef4bb68e1428a943a43f06fb879b8(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:10,530 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:10,530 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/B, priority=13, startTime=1679892970467; duration=0sec 2023-03-27 04:56:10,530 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:10,530 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-27 04:56:10,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:10,531 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-27 04:56:10,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:10,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:10,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:10,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:10,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:10,534 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:10,534 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:10,534 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. because compaction request was cancelled 2023-03-27 04:56:10,542 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:10,545 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/A of e7756c618a32a862ad613327bee74d79 into 97298b673b1a497ca3c72f6746494f5d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:10,546 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:10,546 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/A, priority=13, startTime=1679892970467; duration=0sec 2023-03-27 04:56:10,546 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:10,562 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:10,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:10,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:10,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:10,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:10,570 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:10,570 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:10,570 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:10,570 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:10,571 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,571 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,572 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:10,572 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:10,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:10,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,572 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,572 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:10,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:10,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893030569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:10,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:10,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:10,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:10,574 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:10,574 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:10,574 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:10,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:10,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893030573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:10,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49536 deadline: 1679893030573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:10,578 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:10,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:10,578 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:10,579 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:10,579 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:10,580 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:10,580 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:10,580 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:10,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:10,588 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,588 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,588 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,588 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:10,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74972792[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:10,590 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:10,590 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,591 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,591 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:10,591 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:10,591 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:10,592 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,592 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:10,593 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,593 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:10,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,593 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:10,593 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:10,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:10,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:10,596 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:10,596 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:10,596 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:10,596 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:10,596 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:10,597 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,597 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:10,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,597 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:10,598 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,599 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,601 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,601 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,601 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:10,601 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,601 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:10,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:10,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:10,601 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,601 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,601 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,601 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,601 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:10,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:10,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:10,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:10,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:10,602 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:10,602 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:10,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:10,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:10,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49524 deadline: 1679893030675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:10,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:10,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49536 deadline: 1679893030677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893030677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,702 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:10,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:10,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:10,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:10,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:10,730 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:10,730 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:10,730 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:10,730 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:10,731 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,731 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,731 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:10,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:10,732 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,732 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,732 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:10,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:10,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:10,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:10,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49510 deadline: 1679893030733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:10,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:10,742 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:10,742 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:10,742 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:10,750 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:10,750 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:10,762 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:10,762 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:10,762 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:10,762 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:10,763 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:10,763 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:10,763 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,763 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:10,763 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:10,765 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,765 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,766 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,766 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:10,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:10,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18c52010[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:10,768 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:10,768 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,769 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,769 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:10,769 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:10,769 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:10,769 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,769 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:10,770 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,770 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:10,770 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:10,770 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:10,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:10,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:10,773 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:10,773 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:10,773 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:10,773 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:10,773 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:10,774 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,774 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:10,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,774 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,775 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,776 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,779 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:10,779 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:10,779 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,779 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,779 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,779 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:10,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:10,779 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:10,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:10,779 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,780 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,780 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,780 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:10,780 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:10,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:10,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:10,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:10,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:10,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893030877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,880 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:10,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:10,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:10,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:10,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49536 deadline: 1679893030884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:10,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893030885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:10,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:10,903 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:10,903 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:10,903 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:10,903 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:10,904 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,904 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,905 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:10,905 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:10,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:10,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:10,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:10,906 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,906 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:10,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:10,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:10,907 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:10,907 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:10,907 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:10,908 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:10,908 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:10,908 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:10,908 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:10,908 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:10,908 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:10,908 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:10,908 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:10,908 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,908 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:10,908 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:10,909 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,909 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,910 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,910 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:10,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:10,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31cecf7d[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:10,914 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:10,914 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,915 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,916 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:10,916 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:10,916 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:10,916 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,916 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:10,917 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:10,917 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:10,917 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:10,918 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:10,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:10,920 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:10,920 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:10,920 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:10,920 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:10,920 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:10,931 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,932 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:10,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:10,932 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49552 deadline: 1679893030929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:10,933 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:10,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:10,935 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:10,940 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:10,940 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:10,940 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:10,940 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:10,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:10,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:10,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:10,940 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:10,940 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:10,940 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:10,940 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:10,941 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:10,941 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,941 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:10,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:10,941 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:10,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:10,993 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/f73c4a19c43c49149dba506d5ada5438 2023-03-27 04:56:11,020 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/16511e8dbbb74283a1e80a6e6aef5051 2023-03-27 04:56:11,040 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:11,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:11,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:11,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:11,050 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/abd87f04c5f64272811114046175d748 2023-03-27 04:56:11,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:11,050 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:11,050 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:11,050 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:11,050 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:11,051 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,051 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,052 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:11,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:11,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:11,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,053 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:11,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:11,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:11,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:11,054 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:11,054 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:11,054 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:11,056 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,057 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:11,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/f73c4a19c43c49149dba506d5ada5438 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/f73c4a19c43c49149dba506d5ada5438 2023-03-27 04:56:11,062 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:11,062 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:11,062 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:11,062 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:11,062 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:11,062 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:11,062 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:11,062 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:11,062 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,062 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:11,062 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:11,064 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/f73c4a19c43c49149dba506d5ada5438, entries=150, sequenceid=371, filesize=11.9 K 2023-03-27 04:56:11,065 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,065 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,065 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,065 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:11,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,066 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/16511e8dbbb74283a1e80a6e6aef5051 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/16511e8dbbb74283a1e80a6e6aef5051 2023-03-27 04:56:11,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:11,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dfa6a0[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:11,067 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:11,067 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,068 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:11,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:11,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:11,068 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,068 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:11,069 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,069 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:11,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,069 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:11,069 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:11,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:11,071 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:11,071 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:11,071 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:11,071 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:11,071 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:11,072 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,072 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,072 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:11,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,073 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:11,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:11,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,074 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,075 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:11,075 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,075 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:11,075 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:11,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:11,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:11,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:11,075 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:11,075 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:11,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:11,076 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:11,076 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:11,088 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/16511e8dbbb74283a1e80a6e6aef5051, entries=150, sequenceid=371, filesize=11.9 K 2023-03-27 04:56:11,089 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/abd87f04c5f64272811114046175d748 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/abd87f04c5f64272811114046175d748 2023-03-27 04:56:11,098 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/abd87f04c5f64272811114046175d748, entries=150, sequenceid=371, filesize=11.9 K 2023-03-27 04:56:11,099 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for e7756c618a32a862ad613327bee74d79 in 568ms, sequenceid=371, compaction requested=true 2023-03-27 04:56:11,099 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:11,099 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:11,099 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-27 04:56:11,099 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-27 04:56:11,099 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:11,099 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:11,100 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:11,100 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:11,100 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:11,100 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:11,100 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. because compaction request was cancelled 2023-03-27 04:56:11,100 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. because compaction request was cancelled 2023-03-27 04:56:11,100 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:11,101 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:11,101 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/C is initiating minor compaction (all files) 2023-03-27 04:56:11,101 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/C in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:11,102 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/fea5d17b9cf84edbad27556e8bee3434, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/4ff6928b52a74108943ca02eed9dca48, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/abd87f04c5f64272811114046175d748] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=36.3 K 2023-03-27 04:56:11,102 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting fea5d17b9cf84edbad27556e8bee3434, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679892968787 2023-03-27 04:56:11,102 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 4ff6928b52a74108943ca02eed9dca48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1679892969400 2023-03-27 04:56:11,103 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting abd87f04c5f64272811114046175d748, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1679892969584 2023-03-27 04:56:11,111 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#C#compaction#67 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:11,176 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:11,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:11,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:11,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:11,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:11,179 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:11,179 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:11,179 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:11,179 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:11,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,181 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:11,181 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:11,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:11,182 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,182 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:11,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:11,183 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-27 04:56:11,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:11,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:11,184 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:11,184 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:11,184 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:11,184 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:11,184 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:11,184 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:11,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:11,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:11,184 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:11,184 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:11,184 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:11,184 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:11,185 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:11,185 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:11,185 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:11,185 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:11,185 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:11,185 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:11,185 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:11,185 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,185 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:11,185 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:11,195 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,195 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,196 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,196 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:11,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d39ba9e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:11,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:11,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,207 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,207 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:11,207 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:11,207 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:11,208 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,208 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:11,208 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,208 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:11,208 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:11,209 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:11,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:11,211 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:11,212 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:11,212 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:11,212 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:11,212 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:11,213 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,213 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,213 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:11,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:11,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,215 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,216 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,224 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:11,224 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:11,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:11,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:11,224 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:11,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:11,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:11,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:11,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:11,224 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:11,224 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:11,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:11,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:11,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893031222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:11,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893031223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:11,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49536 deadline: 1679893031225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,230 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,230 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,230 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,230 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,230 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,230 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,230 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,324 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:11,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:11,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:11,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:11,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:11,328 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:11,328 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:11,328 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:11,328 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:11,329 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,329 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,330 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,330 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:11,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:11,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:11,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:11,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893031327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,330 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,330 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:11,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:11,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893031327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:11,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:11,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49536 deadline: 1679893031331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:11,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:11,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:11,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:11,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:11,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:11,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:11,334 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:11,334 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:11,335 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:11,335 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:11,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:11,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:11,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:11,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:11,336 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,336 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,336 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,336 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:11,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ccdd689[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:11,338 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:11,338 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,339 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,339 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:11,339 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:11,339 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:11,340 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,340 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:11,341 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:11,341 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,341 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,341 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:11,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:11,345 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:11,345 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:11,345 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:11,345 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:11,345 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:11,353 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,353 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,353 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,354 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:11,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:11,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,356 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,367 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:11,367 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:11,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:11,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:11,367 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:11,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:11,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:11,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:11,368 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:11,368 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:11,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:11,468 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:11,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:11,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:11,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:11,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:11,474 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:11,474 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:11,474 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:11,474 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:11,478 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,479 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,479 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:11,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:11,480 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,480 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,480 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:11,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:11,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:11,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:11,481 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:11,481 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:11,481 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:11,486 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:11,486 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:11,486 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:11,486 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:11,486 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:11,486 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:11,486 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:11,486 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:11,486 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,486 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:11,487 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:11,498 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,499 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,499 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:11,500 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bbf4b4a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:11,500 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,503 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,503 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:11,503 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:11,503 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:11,503 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,503 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:11,504 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:11,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:11,506 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:11,506 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:11,506 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:11,506 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:11,506 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:11,507 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:11,507 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:11,507 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:11,507 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,507 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,507 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:11,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:11,508 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,509 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,511 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:11,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:11,511 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:11,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:11,514 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,514 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,514 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,514 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,514 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,514 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,514 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,514 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:11,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:11,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:11,514 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:11,514 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:11,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:11,533 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/f1176b8cc9644ad1952ca83d2dc06900 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f1176b8cc9644ad1952ca83d2dc06900 2023-03-27 04:56:11,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893031532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:11,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893031532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:11,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49536 deadline: 1679893031536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,541 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/C of e7756c618a32a862ad613327bee74d79 into f1176b8cc9644ad1952ca83d2dc06900(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:11,541 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:11,541 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/C, priority=13, startTime=1679892971099; duration=0sec 2023-03-27 04:56:11,541 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:11,615 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:11,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:11,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:11,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:11,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:11,622 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:11,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:11,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:11,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:11,623 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,623 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,624 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,624 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:11,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:11,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:11,634 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/8be95bcbcb75465a9483d5b8c4f15e0b 2023-03-27 04:56:11,634 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,634 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:11,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:11,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:11,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:11,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:11,644 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:11,644 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:11,644 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:11,665 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:11,665 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:11,665 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:11,665 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:11,665 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:11,665 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:11,665 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:11,665 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:11,665 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,665 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:11,665 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:11,667 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,667 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,667 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,667 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:11,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46a8103b[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:11,672 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:11,672 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,678 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,678 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:11,678 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:11,678 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:11,679 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,679 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:11,682 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:11,682 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:11,682 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:11,682 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:11,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:11,685 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:11,685 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:11,685 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:11,685 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:11,685 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:11,690 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,690 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,690 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:11,690 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,695 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,701 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,702 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:11,702 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:11,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:11,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:11,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:11,702 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:11,702 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,702 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,702 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:11,702 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:11,702 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:11,702 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:11,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:11,702 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,702 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:11,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:11,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49510 deadline: 1679893031742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,802 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:11,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:11,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:11,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:11,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:11,814 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:11,814 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:11,814 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:11,814 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:11,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,815 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:11,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:11,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:11,816 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,816 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:11,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:11,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:11,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:11,818 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:11,818 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:11,818 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:11,824 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:11,824 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:11,825 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:11,825 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:11,825 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:11,825 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:11,825 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:11,825 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:11,825 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,825 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:11,825 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:11,828 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,828 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:11,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:11,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46c5bf3a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:11,830 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:11,830 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,831 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,831 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:11,831 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:11,831 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:11,832 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,832 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:11,833 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,833 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,833 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:11,833 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893031834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:11,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893031834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:11,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:11,836 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:11,836 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:11,836 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:11,836 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:11,836 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:11,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:11,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49536 deadline: 1679893031839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,845 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,845 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:11,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,845 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,846 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,847 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,853 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,853 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:11,853 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:11,853 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,853 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,853 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:11,853 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,853 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,853 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:11,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:11,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:11,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:11,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:11,854 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:11,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:11,854 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:11,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:11,954 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:11,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:11,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:11,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:11,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:11,957 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:11,957 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:11,957 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:11,957 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:11,958 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,958 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,959 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:11,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:11,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:11,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,959 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,959 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,959 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:11,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:11,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:11,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:11,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:11,960 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:11,960 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:11,960 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:11,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:11,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:11,961 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:11,961 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:11,961 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:11,961 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:11,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:11,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:11,962 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,962 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:11,962 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:11,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,963 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,963 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,963 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:11,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54ae40f5[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:11,964 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:11,964 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,964 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,965 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:11,965 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:11,965 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:11,965 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,965 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:11,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:11,966 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:11,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:11,969 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:11,970 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:11,970 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:11,970 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:11,970 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:11,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:11,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:11,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:11,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:11,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:11,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,982 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,982 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,982 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:11,982 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:11,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:11,982 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:11,982 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,982 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:11,982 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:11,982 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:11,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:11,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:11,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:11,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:11,983 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:11,983 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:11,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:11,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:11,983 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:12,083 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:12,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:12,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:12,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:12,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:12,087 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:12,087 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:12,087 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:12,087 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:12,087 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,087 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,088 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:12,088 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,089 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,089 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:12,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,089 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:12,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:12,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:12,102 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:12,102 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:12,102 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:12,102 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/dc13754af7a44ab48d65af23d927de10 2023-03-27 04:56:12,110 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:12,110 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:12,110 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:12,110 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:12,110 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:12,110 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:12,110 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:12,110 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:12,110 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,110 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:12,110 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:12,111 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,112 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,112 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,112 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:12,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:12,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@150aa431[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:12,113 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:12,114 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,114 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,116 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:12,116 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:12,116 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:12,118 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,118 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:12,119 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:12,119 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,119 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:12,119 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:12,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:12,122 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:12,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:12,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:12,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:12,122 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:12,123 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,123 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:12,123 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,125 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,127 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,131 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:12,131 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:12,131 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,131 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,131 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,131 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,131 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,131 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,131 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:12,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:12,131 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:12,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:12,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:12,132 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:12,132 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:12,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:12,232 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:12,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:12,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:12,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:12,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:12,238 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:12,239 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:12,239 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:12,239 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:12,239 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,239 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,240 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:12,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:12,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:12,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,241 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:12,241 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,241 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:12,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:12,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:12,242 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:12,242 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:12,242 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:12,246 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:12,246 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:12,246 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:12,246 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:12,246 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:12,246 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:12,246 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:12,246 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:12,246 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,246 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:12,246 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:12,256 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,256 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,257 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,257 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13255763[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:12,258 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:12,258 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,270 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,270 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:12,270 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:12,270 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:12,270 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,270 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:12,273 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:12,273 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:12,273 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:12,273 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:12,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:12,276 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:12,276 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:12,276 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:12,276 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:12,276 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:12,278 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,278 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:12,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:12,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,281 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,283 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,284 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:12,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:12,290 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:12,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:12,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:12,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:12,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:12,290 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:12,290 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:12,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:12,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:12,291 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,291 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,291 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,291 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,291 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,291 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,291 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:12,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:12,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893032338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:12,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893032338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:12,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49536 deadline: 1679893032344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,390 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:12,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:12,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:12,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:12,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:12,393 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:12,393 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:12,393 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:12,393 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:12,394 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,394 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,395 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,395 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:12,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:12,395 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,395 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:12,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:12,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:12,396 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:12,396 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:12,396 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:12,396 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:12,396 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:12,396 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:12,396 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:12,396 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:12,396 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:12,396 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:12,396 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:12,396 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,396 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:12,396 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:12,397 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,397 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,397 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,398 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:12,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78f21652[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:12,399 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:12,399 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,399 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,400 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:12,400 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:12,400 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:12,400 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,400 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:12,401 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,402 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:12,402 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:12,402 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:12,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:12,405 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:12,405 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:12,405 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:12,405 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:12,405 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:12,406 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,406 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,406 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:12,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:12,407 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,408 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,409 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,409 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,409 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,409 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,409 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:12,409 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:12,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:12,410 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:12,409 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:12,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:12,410 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,410 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,410 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:12,410 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:12,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:12,510 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:12,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:12,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:12,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:12,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:12,518 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:12,518 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:12,518 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:12,518 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:12,518 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,518 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,519 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,519 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:12,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:12,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:12,519 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,519 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:12,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:12,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:12,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:12,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:12,522 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:12,522 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:12,522 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:12,523 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:12,523 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:12,523 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:12,523 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:12,523 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:12,523 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:12,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:12,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:12,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:12,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:12,526 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,526 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,526 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,527 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:12,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:12,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5efd5851[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:12,529 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:12,529 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,530 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,530 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,530 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:12,530 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:12,530 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:12,530 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:12,531 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,531 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:12,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,531 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:12,531 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:12,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:12,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:12,533 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:12,533 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:12,533 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:12,533 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:12,533 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:12,535 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,535 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:12,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:12,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:12,536 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,536 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,538 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,548 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:12,548 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:12,548 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:12,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:12,548 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,549 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,549 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,549 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,549 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,549 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,549 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:12,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:12,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:12,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:12,549 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:12,549 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:12,574 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/f49ec8bb4469471a94e7298c707554a7 2023-03-27 04:56:12,580 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/8be95bcbcb75465a9483d5b8c4f15e0b as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/8be95bcbcb75465a9483d5b8c4f15e0b 2023-03-27 04:56:12,586 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/8be95bcbcb75465a9483d5b8c4f15e0b, entries=200, sequenceid=390, filesize=14.3 K 2023-03-27 04:56:12,588 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/dc13754af7a44ab48d65af23d927de10 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/dc13754af7a44ab48d65af23d927de10 2023-03-27 04:56:12,595 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/dc13754af7a44ab48d65af23d927de10, entries=150, sequenceid=390, filesize=11.9 K 2023-03-27 04:56:12,598 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/f49ec8bb4469471a94e7298c707554a7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f49ec8bb4469471a94e7298c707554a7 2023-03-27 04:56:12,605 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f49ec8bb4469471a94e7298c707554a7, entries=150, sequenceid=390, filesize=11.9 K 2023-03-27 04:56:12,606 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for e7756c618a32a862ad613327bee74d79 in 1422ms, sequenceid=390, compaction requested=true 2023-03-27 04:56:12,606 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:12,606 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:12,606 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:12,606 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:12,606 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:12,606 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:12,607 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39714 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:12,607 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/A is initiating minor compaction (all files) 2023-03-27 04:56:12,607 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/A in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:12,607 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/97298b673b1a497ca3c72f6746494f5d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/f73c4a19c43c49149dba506d5ada5438, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/8be95bcbcb75465a9483d5b8c4f15e0b] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=38.8 K 2023-03-27 04:56:12,608 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:12,608 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 97298b673b1a497ca3c72f6746494f5d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1679892969400 2023-03-27 04:56:12,608 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/B is initiating minor compaction (all files) 2023-03-27 04:56:12,608 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/B in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:12,608 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/fc1ef4bb68e1428a943a43f06fb879b8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/16511e8dbbb74283a1e80a6e6aef5051, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/dc13754af7a44ab48d65af23d927de10] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=36.4 K 2023-03-27 04:56:12,608 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting f73c4a19c43c49149dba506d5ada5438, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1679892969584 2023-03-27 04:56:12,609 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting fc1ef4bb68e1428a943a43f06fb879b8, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1679892969400 2023-03-27 04:56:12,609 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 8be95bcbcb75465a9483d5b8c4f15e0b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1679892970567 2023-03-27 04:56:12,609 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 16511e8dbbb74283a1e80a6e6aef5051, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1679892969584 2023-03-27 04:56:12,609 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting dc13754af7a44ab48d65af23d927de10, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1679892970570 2023-03-27 04:56:12,628 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#B#compaction#71 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:12,629 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#A#compaction#72 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:12,649 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:12,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:12,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:12,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:12,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:12,653 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:12,653 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:12,653 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:12,653 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:12,654 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,654 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,655 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:12,655 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:12,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:12,655 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,655 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:12,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:12,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:12,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:12,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:12,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:12,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:12,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:12,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:12,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:12,658 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:12,658 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:12,658 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-27 04:56:12,659 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:12,659 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:12,659 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:12,659 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:12,659 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:12,659 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:12,680 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/9013b51b608344dfb98322c79293f639 2023-03-27 04:56:12,721 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/99944657dce2447ea56e9bbd3104bb44 2023-03-27 04:56:12,745 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/72b96b45337d4b93966aaf1ec315320d 2023-03-27 04:56:12,755 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/9013b51b608344dfb98322c79293f639 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/9013b51b608344dfb98322c79293f639 2023-03-27 04:56:12,763 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/9013b51b608344dfb98322c79293f639, entries=150, sequenceid=405, filesize=11.9 K 2023-03-27 04:56:12,765 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/99944657dce2447ea56e9bbd3104bb44 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/99944657dce2447ea56e9bbd3104bb44 2023-03-27 04:56:12,776 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/99944657dce2447ea56e9bbd3104bb44, entries=150, sequenceid=405, filesize=11.9 K 2023-03-27 04:56:12,778 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/72b96b45337d4b93966aaf1ec315320d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/72b96b45337d4b93966aaf1ec315320d 2023-03-27 04:56:12,795 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/72b96b45337d4b93966aaf1ec315320d, entries=150, sequenceid=405, filesize=11.9 K 2023-03-27 04:56:12,797 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for e7756c618a32a862ad613327bee74d79 in 139ms, sequenceid=405, compaction requested=true 2023-03-27 04:56:12,797 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:12,797 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:12,797 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:12,797 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:12,798 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,798 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:12,798 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:12,799 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,799 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,799 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:12,799 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:12,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e164493[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:12,801 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:12,801 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,802 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,802 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:12,802 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:12,802 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:12,803 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,803 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:12,804 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:12,804 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:12,804 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:12,804 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:12,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:12,808 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:12,808 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:12,808 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:12,808 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:12,808 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:12,809 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,809 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,810 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:12,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,811 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,813 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,815 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:12,815 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:12,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:12,816 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,816 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,816 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,816 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,816 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,816 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,816 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:12,816 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:12,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:12,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:12,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:12,817 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:12,817 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:12,917 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:12,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:12,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:12,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:12,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:12,926 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:12,926 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:12,926 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:12,926 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:12,928 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,928 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,929 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:12,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:12,929 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,929 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,929 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:12,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:12,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:12,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:12,938 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:12,938 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:12,938 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:12,946 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:12,946 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:12,954 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:12,954 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:12,954 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:12,954 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:12,954 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:12,954 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:12,954 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,955 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:12,955 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:12,956 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:12,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:12,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2731967d[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:12,959 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:12,959 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,960 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,960 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:12,960 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:12,960 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:12,960 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,960 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:12,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:12,961 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:12,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:12,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:12,966 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:12,966 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:12,966 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:12,966 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:12,966 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:12,967 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,967 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:12,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,967 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:12,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:12,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:12,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:12,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:12,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,968 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:12,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,971 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:12,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,974 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:12,974 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:12,974 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,974 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,974 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:12,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:12,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:12,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:12,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:12,974 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:12,974 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,974 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:12,974 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:12,974 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:12,974 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:12,974 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:12,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:12,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:12,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:12,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:12,983 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:56:12,983 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:12,983 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:12,983 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:12,983 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:12,983 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:12,983 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:13,046 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/6c61cc3dd7b24f6eb4f27481674f1d79 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/6c61cc3dd7b24f6eb4f27481674f1d79 2023-03-27 04:56:13,052 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/B of e7756c618a32a862ad613327bee74d79 into 6c61cc3dd7b24f6eb4f27481674f1d79(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:13,052 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:13,052 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/B, priority=13, startTime=1679892972606; duration=0sec 2023-03-27 04:56:13,052 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:13,052 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:13,054 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:13,054 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/C is initiating minor compaction (all files) 2023-03-27 04:56:13,054 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/C in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:13,055 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f1176b8cc9644ad1952ca83d2dc06900, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f49ec8bb4469471a94e7298c707554a7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/72b96b45337d4b93966aaf1ec315320d] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=36.4 K 2023-03-27 04:56:13,075 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:13,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:13,078 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting f1176b8cc9644ad1952ca83d2dc06900, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1679892969584 2023-03-27 04:56:13,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:13,082 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting f49ec8bb4469471a94e7298c707554a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1679892970570 2023-03-27 04:56:13,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:13,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:13,083 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:13,083 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:13,083 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:13,083 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:13,084 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,084 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,085 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:13,085 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,085 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,085 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:13,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:13,087 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 72b96b45337d4b93966aaf1ec315320d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679892971219 2023-03-27 04:56:13,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:13,092 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:13,092 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:13,092 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:13,093 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/29cbf7aa65164963b6f229b1c2bcb6d6 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/29cbf7aa65164963b6f229b1c2bcb6d6 2023-03-27 04:56:13,094 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:13,094 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:13,094 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:13,094 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:13,094 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:13,095 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:13,095 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:13,095 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:13,095 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,095 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:13,095 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:13,098 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,098 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,098 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,099 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:13,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679892973027, max timestamp=1679892973096Num uniques 859; 2023-03-27 04:56:13,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-27 04:56:13,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679892973027, max timestamp=1679892973096Num uniques 859; 2023-03-27 04:56:13,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-27 04:56:13,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679892973027, max timestamp=1679892973096Num uniques 859; 2023-03-27 04:56:13,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-27 04:56:13,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:13,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f99444f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:13,102 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:13,102 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,103 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/A of e7756c618a32a862ad613327bee74d79 into 29cbf7aa65164963b6f229b1c2bcb6d6(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:13,103 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:13,103 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/A, priority=13, startTime=1679892972606; duration=0sec 2023-03-27 04:56:13,103 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:13,108 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#C#compaction#77 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:13,115 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,116 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:13,116 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:13,116 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:13,116 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,116 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:13,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:13,120 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:13,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:13,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:13,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:13,122 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:13,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:13,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:13,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:13,122 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:13,123 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,124 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,124 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:13,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:13,125 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,126 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,136 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:13,136 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,136 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,136 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,136 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,136 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,137 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,137 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:13,137 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:13,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:13,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:13,136 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:13,137 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:13,137 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:13,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:13,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:13,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893033205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,237 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:13,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:13,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:13,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:13,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:13,245 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:13,245 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:13,245 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:13,245 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:13,245 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,245 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,247 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:13,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:13,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:13,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:13,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:13,250 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:13,250 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:13,250 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:13,258 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:13,258 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:13,258 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:13,258 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:13,258 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:13,258 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:13,258 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:13,258 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:13,258 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,258 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:13,258 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:13,262 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,262 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,262 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:13,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b92a23d[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:13,263 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:13,263 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,264 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:13,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:13,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:13,264 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,264 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:13,265 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,265 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:13,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,265 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:13,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,265 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:13,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:13,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:13,267 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:13,268 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:13,268 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:13,268 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:13,268 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:13,269 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,269 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,269 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:13,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:13,270 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,271 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,272 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,272 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:13,272 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:13,272 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,273 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:13,273 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:13,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:13,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:13,273 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:13,273 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,273 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,273 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,273 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:13,273 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:13,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:13,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:13,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:13,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:13,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893033312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:13,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893033343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:13,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49500 deadline: 1679893033343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:13,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49536 deadline: 1679893033354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,373 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:13,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:13,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:13,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:13,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:13,382 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:13,382 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:13,382 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:13,382 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:13,383 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,383 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,383 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:13,383 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:13,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:13,384 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,384 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,385 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:13,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:13,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:13,390 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:13,390 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:13,390 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:13,390 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:13,390 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:13,390 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:13,391 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:13,391 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:13,391 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:13,391 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:13,391 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:13,391 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,391 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:13,391 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:13,398 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,398 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:13,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@570c4300[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:13,400 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:13,400 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,403 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:13,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:13,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:13,404 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,404 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:13,405 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,406 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:13,406 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:13,406 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:13,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:13,418 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:13,418 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:13,418 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:13,418 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:13,418 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:13,420 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,420 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:13,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:13,421 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,421 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/dc6d14ef1cdc406dbd5c8ad91d7dedd5 2023-03-27 04:56:13,420 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,425 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,436 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,437 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:13,437 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:13,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:13,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:13,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:13,437 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:13,437 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:13,437 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,437 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,437 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,437 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,437 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,437 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:13,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:13,437 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:13,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:13,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:13,449 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/6f3b43f27a3f4a429f027668ff52e5ab 2023-03-27 04:56:13,477 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/5c8d45ccb1f64729a4caca3f298ae098 2023-03-27 04:56:13,485 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/dc6d14ef1cdc406dbd5c8ad91d7dedd5 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/dc6d14ef1cdc406dbd5c8ad91d7dedd5 2023-03-27 04:56:13,495 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/dc6d14ef1cdc406dbd5c8ad91d7dedd5, entries=250, sequenceid=416, filesize=16.7 K 2023-03-27 04:56:13,498 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/6f3b43f27a3f4a429f027668ff52e5ab as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/6f3b43f27a3f4a429f027668ff52e5ab 2023-03-27 04:56:13,504 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/6f3b43f27a3f4a429f027668ff52e5ab, entries=150, sequenceid=416, filesize=11.9 K 2023-03-27 04:56:13,505 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/5c8d45ccb1f64729a4caca3f298ae098 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5c8d45ccb1f64729a4caca3f298ae098 2023-03-27 04:56:13,511 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5c8d45ccb1f64729a4caca3f298ae098, entries=150, sequenceid=416, filesize=11.9 K 2023-03-27 04:56:13,512 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e7756c618a32a862ad613327bee74d79 in 530ms, sequenceid=416, compaction requested=true 2023-03-27 04:56:13,513 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:13,513 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:13,513 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:13,513 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:13,513 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-27 04:56:13,514 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42256 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:13,514 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/A is initiating minor compaction (all files) 2023-03-27 04:56:13,514 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/A in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:13,515 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/29cbf7aa65164963b6f229b1c2bcb6d6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/9013b51b608344dfb98322c79293f639, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/dc6d14ef1cdc406dbd5c8ad91d7dedd5] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=41.3 K 2023-03-27 04:56:13,515 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 29cbf7aa65164963b6f229b1c2bcb6d6, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1679892970570 2023-03-27 04:56:13,516 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 9013b51b608344dfb98322c79293f639, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679892971219 2023-03-27 04:56:13,516 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting dc6d14ef1cdc406dbd5c8ad91d7dedd5, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1679892972962 2023-03-27 04:56:13,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:13,522 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-03-27 04:56:13,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:13,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:13,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:13,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:13,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:13,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:13,526 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#A#compaction#80 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:13,537 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:13,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:13,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:13,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:13,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:13,547 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:13,547 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:13,547 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:13,547 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:13,547 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,547 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,548 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:13,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:13,553 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,553 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,553 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:13,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:13,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893033554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:13,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:13,566 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:13,566 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:13,566 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:13,566 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:13,566 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:13,567 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:13,567 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:13,567 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:13,567 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:13,567 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:13,567 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:13,568 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,568 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:13,568 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:13,569 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,569 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,569 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:13,570 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,576 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/20a36c5ba87a4ffba6f258f84f83f4fa as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/20a36c5ba87a4ffba6f258f84f83f4fa 2023-03-27 04:56:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:13,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca5a5e2[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:13,576 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:13,576 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,582 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,582 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,582 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:13,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,582 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:13,582 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:13,582 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:13,585 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/51100602df744299ac78f0bac5740855 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/51100602df744299ac78f0bac5740855 2023-03-27 04:56:13,586 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/C of e7756c618a32a862ad613327bee74d79 into 20a36c5ba87a4ffba6f258f84f83f4fa(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:13,586 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:13,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:13,587 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:13,587 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:13,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,586 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/C, priority=13, startTime=1679892972606; duration=0sec 2023-03-27 04:56:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,587 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-27 04:56:13,587 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:13,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:13,590 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:13,590 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:13,590 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:13,590 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:13,590 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:13,593 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:13,593 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/B is initiating minor compaction (all files) 2023-03-27 04:56:13,593 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/B in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:13,593 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/6c61cc3dd7b24f6eb4f27481674f1d79, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/99944657dce2447ea56e9bbd3104bb44, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/6f3b43f27a3f4a429f027668ff52e5ab] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=36.5 K 2023-03-27 04:56:13,594 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 6c61cc3dd7b24f6eb4f27481674f1d79, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1679892970570 2023-03-27 04:56:13,594 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 99944657dce2447ea56e9bbd3104bb44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679892971219 2023-03-27 04:56:13,595 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 6f3b43f27a3f4a429f027668ff52e5ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1679892972962 2023-03-27 04:56:13,597 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/A of e7756c618a32a862ad613327bee74d79 into 51100602df744299ac78f0bac5740855(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:13,597 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:13,597 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/A, priority=13, startTime=1679892973513; duration=0sec 2023-03-27 04:56:13,597 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:13,597 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-27 04:56:13,598 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,598 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,598 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:13,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:13,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:13,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,599 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,600 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:13,600 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:13,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,600 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. because compaction request was cancelled 2023-03-27 04:56:13,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,601 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:13,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:13,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:13,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,607 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:13,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:13,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:13,608 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,608 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,608 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,608 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,608 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:13,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:13,609 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:13,609 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:13,611 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#B#compaction#82 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:13,644 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/31773a73379f41538f83453d62ab8926 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/31773a73379f41538f83453d62ab8926 2023-03-27 04:56:13,650 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/B of e7756c618a32a862ad613327bee74d79 into 31773a73379f41538f83453d62ab8926(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:13,651 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:13,651 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/B, priority=13, startTime=1679892973513; duration=0sec 2023-03-27 04:56:13,651 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:13,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:13,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893033660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,709 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:13,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:13,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:13,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:13,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:13,713 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:13,713 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:13,713 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:13,714 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:13,714 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,714 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,715 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:13,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:13,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:13,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:13,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:13,717 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:13,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:13,717 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:13,717 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:13,717 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:13,721 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,722 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:13,722 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:13,722 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:13,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:13,722 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:13,722 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:13,722 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:13,722 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:13,722 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:13,722 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,722 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:13,722 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:13,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,724 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,724 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,724 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:13,725 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:13,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@799b3d05[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:13,725 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,726 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,726 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:13,726 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,726 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:13,726 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:13,726 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:13,728 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:13,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:13,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:13,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:13,734 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:13,734 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:13,734 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:13,734 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:13,734 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:13,737 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,737 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:13,737 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,738 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,740 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,743 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:13,743 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:13,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:13,743 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:13,743 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,743 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,743 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:13,743 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,743 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:13,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:13,743 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:13,743 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:13,744 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:13,744 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:13,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:13,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49510 deadline: 1679893033752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,756 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., hostname=jenkins-hbase20.apache.org,44923,1679892955505, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:56:13,844 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:13,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:13,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:13,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:13,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:13,854 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:13,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:13,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:13,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:13,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,855 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,855 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:13,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:13,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:13,855 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,855 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:13,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:13,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:13,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:13,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:13,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:13,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:13,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:13,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:13,857 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:13,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:13,857 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:13,857 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:13,857 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:13,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:13,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:13,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:13,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:13,859 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,860 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,861 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,861 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:13,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:13,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:13,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17f6fb41[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:13,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,863 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,863 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:13,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:13,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:13,864 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,864 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:13,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:13,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:13,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893033865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:13,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:13,868 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:13,868 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:13,868 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:13,868 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:13,868 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:13,870 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,870 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:13,870 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:13,870 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:13,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,873 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:13,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,874 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,874 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,874 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,874 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:13,874 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:13,874 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:13,874 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:13,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:13,874 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:13,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:13,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,875 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:13,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:13,875 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:13,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:13,975 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:13,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:13,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:13,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:13,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:13,978 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:13,978 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:13,978 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:13,978 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:13,978 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,978 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,979 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:13,979 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:13,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:13,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:13,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:13,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:13,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:13,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:13,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:13,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:13,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:13,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:13,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:13,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:13,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:13,988 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:13,988 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:13,988 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/7088d1cdc62e4710b414e88a1ab62bf2 2023-03-27 04:56:13,988 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:13,989 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:13,989 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:13,989 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:13,989 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,989 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:13,989 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:13,992 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:13,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:13,993 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:13,993 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:13,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:13,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:13,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:13,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:13,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:14,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2257befc[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:14,005 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:14,005 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,020 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,020 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:14,020 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:14,020 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:14,020 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,020 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:14,021 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,021 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,022 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:14,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,022 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:14,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:14,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:14,024 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:14,024 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:14,024 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:14,024 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:14,024 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:14,037 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,037 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:14,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,037 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:14,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:14,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,039 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,040 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,047 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:14,047 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:14,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:14,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:14,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:14,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:14,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:14,048 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:14,048 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,048 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:14,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:14,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:14,048 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:14,048 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,048 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,049 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,049 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,049 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,049 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,049 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/137f6d525970442aa8d94fa40c5851e4 2023-03-27 04:56:14,148 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:14,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:14,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:14,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:14,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:14,152 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:14,152 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:14,152 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:14,152 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:14,152 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,152 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,153 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,153 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:14,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:14,153 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,153 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:14,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:14,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:14,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:14,156 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:14,156 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:14,156 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:14,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:14,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:14,158 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:14,158 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:14,158 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:14,158 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:14,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:14,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:14,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:14,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:14,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:14,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893034168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,173 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,173 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,174 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,174 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:14,175 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:14,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@730c85e1[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:14,175 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,176 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:14,176 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,176 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:14,176 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:14,176 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:14,178 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:14,178 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,178 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:14,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,178 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:14,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:14,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:14,180 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:14,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:14,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:14,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:14,180 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:14,183 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,183 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,183 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,184 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,186 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,192 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:14,192 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:14,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:14,193 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:14,193 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,193 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:14,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:14,193 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,193 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,193 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,193 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:14,193 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:14,193 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,193 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:14,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:14,293 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:14,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:14,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:14,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:14,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:14,304 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:14,304 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:14,305 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:14,305 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:14,305 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,305 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,306 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:14,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:14,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:14,309 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,310 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,310 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:14,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:14,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:14,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:14,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:14,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:14,324 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:14,325 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:14,325 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:14,325 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:14,325 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:14,325 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:14,325 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:14,325 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:14,325 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,325 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:14,325 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:14,326 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,326 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,327 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,327 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:14,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:14,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53dffd90[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:14,331 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:14,331 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,332 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,333 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:14,333 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:14,333 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:14,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:14,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:14,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:14,334 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,334 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:14,334 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:14,334 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:14,335 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:14,335 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:14,335 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:14,335 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:14,336 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,336 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:14,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,336 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,338 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,342 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,353 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,353 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:14,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:14,353 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,353 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,354 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,353 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:14,354 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:14,354 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:14,353 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:14,354 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:14,354 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:14,354 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:14,454 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:14,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:14,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:14,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:14,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:14,457 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:14,457 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:14,457 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:14,457 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:14,458 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,459 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:14,459 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:14,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:14,460 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,460 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:14,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:14,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:14,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:14,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:14,461 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:14,461 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:14,461 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:14,461 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:14,461 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:14,461 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:14,461 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:14,461 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:14,461 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:14,461 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:14,461 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:14,461 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,461 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:14,462 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:14,462 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,462 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,462 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,463 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:14,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:14,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73d32859[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:14,464 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:14,464 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,464 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,465 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:14,465 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:14,465 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:14,465 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,465 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:14,466 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:14,466 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:14,466 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:14,466 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:14,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:14,468 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:14,468 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:14,468 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:14,468 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:14,468 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:14,469 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,469 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:14,469 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,470 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,471 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,472 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/c4d6a92436c0435d88b5a81b4b6a2529 2023-03-27 04:56:14,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,473 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:14,473 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:14,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:14,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:14,473 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,473 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,473 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,473 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,474 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,474 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,474 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,474 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:14,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:14,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:14,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:14,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:14,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:14,474 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:14,474 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:14,477 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/7088d1cdc62e4710b414e88a1ab62bf2 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/7088d1cdc62e4710b414e88a1ab62bf2 2023-03-27 04:56:14,482 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/7088d1cdc62e4710b414e88a1ab62bf2, entries=200, sequenceid=454, filesize=14.3 K 2023-03-27 04:56:14,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/137f6d525970442aa8d94fa40c5851e4 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/137f6d525970442aa8d94fa40c5851e4 2023-03-27 04:56:14,502 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/137f6d525970442aa8d94fa40c5851e4, entries=150, sequenceid=454, filesize=11.9 K 2023-03-27 04:56:14,503 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/c4d6a92436c0435d88b5a81b4b6a2529 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/c4d6a92436c0435d88b5a81b4b6a2529 2023-03-27 04:56:14,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/c4d6a92436c0435d88b5a81b4b6a2529, entries=150, sequenceid=454, filesize=11.9 K 2023-03-27 04:56:14,514 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for e7756c618a32a862ad613327bee74d79 in 992ms, sequenceid=454, compaction requested=true 2023-03-27 04:56:14,514 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:14,514 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:14,514 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-27 04:56:14,514 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-27 04:56:14,514 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:14,515 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:14,515 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:14,516 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:14,516 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. because compaction request was cancelled 2023-03-27 04:56:14,516 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:14,516 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:14,516 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:14,516 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. because compaction request was cancelled 2023-03-27 04:56:14,518 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:14,519 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/C is initiating minor compaction (all files) 2023-03-27 04:56:14,519 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/C in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:14,519 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/20a36c5ba87a4ffba6f258f84f83f4fa, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5c8d45ccb1f64729a4caca3f298ae098, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/c4d6a92436c0435d88b5a81b4b6a2529] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=36.5 K 2023-03-27 04:56:14,520 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 20a36c5ba87a4ffba6f258f84f83f4fa, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679892971219 2023-03-27 04:56:14,520 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 5c8d45ccb1f64729a4caca3f298ae098, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1679892972962 2023-03-27 04:56:14,521 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting c4d6a92436c0435d88b5a81b4b6a2529, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679892973149 2023-03-27 04:56:14,531 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#C#compaction#85 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:14,574 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:14,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:14,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:14,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:14,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:14,578 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:14,578 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:14,578 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:14,578 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:14,578 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,579 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,579 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:14,579 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,580 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,580 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:14,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:14,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:14,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:14,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:14,588 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:14,588 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:14,589 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:14,589 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:14,589 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:14,589 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:14,589 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:14,589 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-27 04:56:14,589 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:14,589 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:14,590 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:14,590 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:14,590 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:14,590 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:14,605 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/e02fc3ca85124b8cb4f20029d003a41c 2023-03-27 04:56:14,620 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/aaa5bac5e49d430a95a124aa6e58fc77 2023-03-27 04:56:14,634 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/548a54be9b2a457e8134e8c6e5c4322d 2023-03-27 04:56:14,640 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/e02fc3ca85124b8cb4f20029d003a41c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/e02fc3ca85124b8cb4f20029d003a41c 2023-03-27 04:56:14,645 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/e02fc3ca85124b8cb4f20029d003a41c, entries=150, sequenceid=466, filesize=11.9 K 2023-03-27 04:56:14,647 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/aaa5bac5e49d430a95a124aa6e58fc77 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/aaa5bac5e49d430a95a124aa6e58fc77 2023-03-27 04:56:14,652 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/aaa5bac5e49d430a95a124aa6e58fc77, entries=150, sequenceid=466, filesize=11.9 K 2023-03-27 04:56:14,653 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/548a54be9b2a457e8134e8c6e5c4322d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/548a54be9b2a457e8134e8c6e5c4322d 2023-03-27 04:56:14,658 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/548a54be9b2a457e8134e8c6e5c4322d, entries=150, sequenceid=466, filesize=11.9 K 2023-03-27 04:56:14,659 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for e7756c618a32a862ad613327bee74d79 in 70ms, sequenceid=466, compaction requested=true 2023-03-27 04:56:14,659 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:14,659 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:14,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:14,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:14,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:14,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:14,661 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:14,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cecd881[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:14,663 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:14,663 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,663 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,663 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:14,663 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:14,663 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:14,663 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,664 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:14,664 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,664 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:14,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,664 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:14,664 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:14,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:14,667 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:14,667 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:14,667 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:14,667 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:14,667 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:14,669 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,669 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:14,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,670 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,670 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,671 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,674 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:14,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:14,674 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:14,674 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:14,674 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:14,674 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:14,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:14,674 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,674 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:14,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:14,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:14,674 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,675 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,675 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,675 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,675 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:14,692 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:56:14,692 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:14,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:14,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:14,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:14,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:14,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:14,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/12332f5d09cc4333afa6ea574360a1b4 2023-03-27 04:56:14,720 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/5854badf28184cfa877f6198ea4da89a 2023-03-27 04:56:14,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679892974750, max timestamp=1679892974763Num uniques 859; 2023-03-27 04:56:14,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:14,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679892974750, max timestamp=1679892974763Num uniques 859; 2023-03-27 04:56:14,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:14,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679892974750, max timestamp=1679892974763Num uniques 859; 2023-03-27 04:56:14,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:14,774 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:14,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:14,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:14,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:14,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:14,786 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:14,786 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:14,786 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:14,786 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:14,787 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,787 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,787 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:14,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:14,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:14,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:14,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:14,789 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,789 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,789 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:14,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:14,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:14,798 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:14,798 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:14,798 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:14,806 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:14,806 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:14,806 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:14,806 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:14,806 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:14,806 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:14,806 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:14,806 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:14,807 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,807 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:14,807 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:14,811 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,812 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,812 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,812 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:14,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b586b70[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:14,813 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:14,814 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,814 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,814 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:14,814 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:14,814 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:14,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:14,816 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,816 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:14,816 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:14,816 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:14,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:14,819 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:14,819 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:14,819 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:14,819 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:14,819 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:14,820 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,820 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:14,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,820 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,830 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,833 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,834 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:14,834 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:14,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:14,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:14,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:14,834 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,835 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,835 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,835 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,835 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,835 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,834 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:14,835 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:14,835 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:14,834 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:14,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:14,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:14,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893034846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,935 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:14,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:14,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:14,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:14,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:14,938 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:14,939 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:14,939 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:14,939 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:14,939 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,939 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,940 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:14,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:14,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:14,940 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,941 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,941 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:14,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:14,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:14,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:14,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:14,946 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:14,946 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:14,946 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:14,948 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:14,949 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:14,949 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:14,949 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:14,949 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:14,949 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:14,949 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/56312b5b0a7f4acc87dbc3d74377ec5c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/56312b5b0a7f4acc87dbc3d74377ec5c 2023-03-27 04:56:14,949 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:14,949 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:14,949 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,949 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:14,949 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:14,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:14,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893034951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,956 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,957 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,957 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,957 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:14,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:14,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57c72af6[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:14,960 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:14,960 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,961 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/C of e7756c618a32a862ad613327bee74d79 into 56312b5b0a7f4acc87dbc3d74377ec5c(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:14,961 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:14,961 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/C, priority=13, startTime=1679892974515; duration=0sec 2023-03-27 04:56:14,961 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:14,965 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,965 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:14,965 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:14,965 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:14,965 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,965 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:14,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:14,966 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:14,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:14,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:14,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:14,970 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:14,970 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:14,970 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:14,970 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:14,970 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:14,972 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,972 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:14,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,972 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:14,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:14,973 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:14,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,975 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:14,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:14,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,977 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,977 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,977 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:14,977 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:14,977 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:14,977 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:14,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:14,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:14,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:14,977 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:14,977 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:14,978 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:14,978 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:14,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:14,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:15,078 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:15,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:15,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:15,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:15,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:15,085 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:15,085 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:15,085 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:15,085 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:15,085 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,085 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,086 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,086 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:15,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:15,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:15,086 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,087 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:15,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:15,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:15,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:15,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:15,088 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:15,088 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:15,088 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:15,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:15,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:15,090 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:15,090 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:15,090 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:15,090 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:15,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:15,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:15,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,091 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:15,091 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:15,095 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,095 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,095 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,095 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:15,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:15,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dbc41b7[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:15,097 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:15,097 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,098 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,098 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:15,098 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:15,098 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:15,098 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,098 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:15,099 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,099 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:15,099 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:15,099 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:15,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:15,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:15,104 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:15,104 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:15,104 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:15,104 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:15,104 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:15,111 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,111 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:15,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,119 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,120 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,123 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:15,123 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:15,123 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,123 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,123 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,123 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,123 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,123 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:15,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:15,123 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:15,123 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:15,123 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:15,123 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:15,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:15,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:15,143 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/7e17ccd1844745218f5ebb28a639ad69 2023-03-27 04:56:15,157 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/12332f5d09cc4333afa6ea574360a1b4 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/12332f5d09cc4333afa6ea574360a1b4 2023-03-27 04:56:15,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:15,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893035153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,163 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/12332f5d09cc4333afa6ea574360a1b4, entries=200, sequenceid=476, filesize=14.3 K 2023-03-27 04:56:15,164 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/5854badf28184cfa877f6198ea4da89a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/5854badf28184cfa877f6198ea4da89a 2023-03-27 04:56:15,169 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/5854badf28184cfa877f6198ea4da89a, entries=150, sequenceid=476, filesize=11.9 K 2023-03-27 04:56:15,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/7e17ccd1844745218f5ebb28a639ad69 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/7e17ccd1844745218f5ebb28a639ad69 2023-03-27 04:56:15,187 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/7e17ccd1844745218f5ebb28a639ad69, entries=150, sequenceid=476, filesize=11.9 K 2023-03-27 04:56:15,188 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e7756c618a32a862ad613327bee74d79 in 496ms, sequenceid=476, compaction requested=true 2023-03-27 04:56:15,188 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:15,188 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:56:15,188 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:15,189 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:15,189 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-27 04:56:15,189 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:56:15,206 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:56:15,206 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:56:15,207 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/B is initiating minor compaction (all files) 2023-03-27 04:56:15,207 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/A is initiating minor compaction (all files) 2023-03-27 04:56:15,207 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/B in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:15,207 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/A in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:15,207 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/31773a73379f41538f83453d62ab8926, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/137f6d525970442aa8d94fa40c5851e4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/aaa5bac5e49d430a95a124aa6e58fc77, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/5854badf28184cfa877f6198ea4da89a] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=48.5 K 2023-03-27 04:56:15,207 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/51100602df744299ac78f0bac5740855, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/7088d1cdc62e4710b414e88a1ab62bf2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/e02fc3ca85124b8cb4f20029d003a41c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/12332f5d09cc4333afa6ea574360a1b4] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=53.3 K 2023-03-27 04:56:15,207 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 31773a73379f41538f83453d62ab8926, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1679892972962 2023-03-27 04:56:15,208 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 51100602df744299ac78f0bac5740855, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1679892972962 2023-03-27 04:56:15,208 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 137f6d525970442aa8d94fa40c5851e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679892973149 2023-03-27 04:56:15,208 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 7088d1cdc62e4710b414e88a1ab62bf2, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679892973149 2023-03-27 04:56:15,208 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting aaa5bac5e49d430a95a124aa6e58fc77, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1679892973531 2023-03-27 04:56:15,209 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting e02fc3ca85124b8cb4f20029d003a41c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1679892973531 2023-03-27 04:56:15,209 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 5854badf28184cfa877f6198ea4da89a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1679892974679 2023-03-27 04:56:15,209 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 12332f5d09cc4333afa6ea574360a1b4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1679892974679 2023-03-27 04:56:15,222 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#B#compaction#93 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:15,222 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#A#compaction#92 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:15,224 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:15,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:15,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:15,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:15,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:15,233 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:15,233 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:15,233 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:15,233 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:15,233 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,233 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,234 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:15,234 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:15,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:15,234 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,234 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:15,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:15,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:15,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:15,239 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:15,239 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:15,239 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:15,242 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:15,242 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:15,242 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:15,242 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:15,243 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-27 04:56:15,243 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:15,243 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:15,243 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:15,243 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:15,243 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:15,243 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:15,256 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/767430fda6a546a2855522ecba91f372 2023-03-27 04:56:15,269 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/61514612cbf74eaa8830c0147e6d4ff8 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/61514612cbf74eaa8830c0147e6d4ff8 2023-03-27 04:56:15,276 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e7756c618a32a862ad613327bee74d79/B of e7756c618a32a862ad613327bee74d79 into 61514612cbf74eaa8830c0147e6d4ff8(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:15,276 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:15,276 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/B, priority=12, startTime=1679892975189; duration=0sec 2023-03-27 04:56:15,276 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:15,276 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:15,278 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:15,278 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/C is initiating minor compaction (all files) 2023-03-27 04:56:15,278 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/C in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:15,278 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/56312b5b0a7f4acc87dbc3d74377ec5c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/548a54be9b2a457e8134e8c6e5c4322d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/7e17ccd1844745218f5ebb28a639ad69] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=36.6 K 2023-03-27 04:56:15,279 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 56312b5b0a7f4acc87dbc3d74377ec5c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679892973149 2023-03-27 04:56:15,279 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 548a54be9b2a457e8134e8c6e5c4322d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1679892973531 2023-03-27 04:56:15,280 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 7e17ccd1844745218f5ebb28a639ad69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1679892974679 2023-03-27 04:56:15,315 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#C#compaction#96 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:15,320 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/0166819a3fbb4d65a85baf2bfdbd4bc8 2023-03-27 04:56:15,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:15,352 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:15,357 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/b3209cb8a8464522bb6bbc2e62b51494 2023-03-27 04:56:15,365 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/767430fda6a546a2855522ecba91f372 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/767430fda6a546a2855522ecba91f372 2023-03-27 04:56:15,369 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/767430fda6a546a2855522ecba91f372, entries=150, sequenceid=514, filesize=11.9 K 2023-03-27 04:56:15,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:15,370 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/0166819a3fbb4d65a85baf2bfdbd4bc8 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/0166819a3fbb4d65a85baf2bfdbd4bc8 2023-03-27 04:56:15,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49500 deadline: 1679893035366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:15,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893035371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:15,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49536 deadline: 1679893035373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,376 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., hostname=jenkins-hbase20.apache.org,44923,1679892955505, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:56:15,381 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/0166819a3fbb4d65a85baf2bfdbd4bc8, entries=150, sequenceid=514, filesize=11.9 K 2023-03-27 04:56:15,382 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/b3209cb8a8464522bb6bbc2e62b51494 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/b3209cb8a8464522bb6bbc2e62b51494 2023-03-27 04:56:15,388 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/b3209cb8a8464522bb6bbc2e62b51494, entries=150, sequenceid=514, filesize=11.9 K 2023-03-27 04:56:15,389 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for e7756c618a32a862ad613327bee74d79 in 147ms, sequenceid=514, compaction requested=false 2023-03-27 04:56:15,389 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:15,389 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:15,389 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:15,389 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:15,389 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,389 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:15,389 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:15,399 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:15,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@371ebe51[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:15,401 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:15,401 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,401 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,402 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:15,402 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:15,402 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:15,402 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,402 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:15,402 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:15,403 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:15,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:15,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:15,405 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:15,405 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:15,405 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:15,406 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:15,406 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:15,407 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,407 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:15,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,407 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:15,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,408 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,411 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,413 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:15,413 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:15,413 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,413 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,413 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,413 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:15,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:15,413 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,413 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:15,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:15,414 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:15,413 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:15,414 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:15,413 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:15,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:15,464 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-27 04:56:15,464 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:15,464 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:15,464 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:15,464 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:15,464 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:15,464 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:15,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679892975491, max timestamp=1679892975499Num uniques 859; 2023-03-27 04:56:15,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:15,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679892975491, max timestamp=1679892975499Num uniques 859; 2023-03-27 04:56:15,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679892975491, max timestamp=1679892975499Num uniques 859; 2023-03-27 04:56:15,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:15,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:15,514 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:15,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:15,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:15,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:15,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:15,518 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:15,518 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:15,518 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:15,518 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:15,518 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,518 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,519 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:15,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:15,519 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:15,519 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,519 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:15,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:15,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:15,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:15,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:15,520 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:15,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:15,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:15,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:15,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:15,521 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:15,521 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:15,521 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:15,521 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:15,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:15,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:15,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:15,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:15,522 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,522 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,523 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,523 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:15,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:15,524 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:15,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fa54029[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:15,524 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,525 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,525 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,525 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:15,525 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:15,525 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:15,525 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:15,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:15,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49500 deadline: 1679893035519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,526 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:15,526 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:15,526 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,526 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893035522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:15,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:15,528 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:15,528 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:15,528 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:15,528 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:15,528 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:15,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:15,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893035526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,530 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,530 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,530 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,530 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,531 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,533 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:15,533 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:15,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:15,533 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:15,533 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:15,533 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,533 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,533 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,533 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,533 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:15,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:15,534 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:15,533 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:15,534 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:15,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:15,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:15,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49500 deadline: 1679893035626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:15,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893035629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:15,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893035630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,634 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:15,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:15,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:15,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:15,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:15,638 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:15,638 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:15,638 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:15,638 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:15,639 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,639 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,639 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:15,639 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:15,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:15,640 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,640 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:15,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:15,652 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/b0aa4815186c454ca2b6ebaea400505e as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/b0aa4815186c454ca2b6ebaea400505e 2023-03-27 04:56:15,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:15,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:15,653 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:15,653 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:15,653 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:15,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:15,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:15,659 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:15,659 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:15,659 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:15,659 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:15,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:15,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:15,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:15,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:15,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,661 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:15,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61db4bb1[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:15,663 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:15,663 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,664 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,664 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e7756c618a32a862ad613327bee74d79/A of e7756c618a32a862ad613327bee74d79 into b0aa4815186c454ca2b6ebaea400505e(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:15,664 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:15,664 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/A, priority=12, startTime=1679892975188; duration=0sec 2023-03-27 04:56:15,664 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:15,664 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,664 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:15,664 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:15,664 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:15,664 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:15,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,665 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:15,665 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:15,665 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:15,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:15,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:15,670 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:15,670 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:15,670 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:15,670 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:15,670 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:15,673 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,673 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:15,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,673 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:15,674 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,676 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,686 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:15,687 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:15,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:15,687 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:15,687 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,687 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:15,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:15,687 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,687 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,687 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,687 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,687 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:15,687 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:15,687 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:15,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:15,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:15,749 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/5c86f7d048864abd912b4353e01a5afc as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5c86f7d048864abd912b4353e01a5afc 2023-03-27 04:56:15,754 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/C of e7756c618a32a862ad613327bee74d79 into 5c86f7d048864abd912b4353e01a5afc(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:15,754 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:15,754 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/C, priority=13, startTime=1679892975189; duration=0sec 2023-03-27 04:56:15,754 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:15,787 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:15,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:15,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:15,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:15,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:15,798 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:15,798 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:15,798 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:15,798 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:15,799 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,799 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,800 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,800 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:15,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:15,801 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,801 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:15,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:15,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:15,809 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:15,809 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:15,809 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:15,809 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:15,809 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:15,809 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:15,809 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:15,809 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:15,809 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:15,809 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:15,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:15,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:15,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:15,811 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,811 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,812 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,812 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:15,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eb51fc4[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:15,813 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:15,813 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,814 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,814 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,814 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:15,814 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:15,814 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:15,814 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:15,815 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,816 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:15,816 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:15,816 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:15,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:15,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:15,818 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:15,818 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:15,818 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:15,818 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:15,818 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:15,819 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,819 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,819 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:15,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:15,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:15,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,820 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,822 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,823 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,823 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:15,823 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,823 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,824 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,824 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:15,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:15,823 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:15,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:15,824 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,824 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:15,824 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:15,824 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:15,824 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:15,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:15,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:15,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:15,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:15,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893035832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:15,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893035836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:15,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49500 deadline: 1679893035836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,872 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/53a770a64b47427c8d92058fdc13d2f6 2023-03-27 04:56:15,896 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/f1913101cefc47df91521d2c94ecd23f 2023-03-27 04:56:15,916 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/fbfbfa3826b84b73858bf4ff27ec74e1 2023-03-27 04:56:15,921 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/53a770a64b47427c8d92058fdc13d2f6 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/53a770a64b47427c8d92058fdc13d2f6 2023-03-27 04:56:15,924 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:15,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:15,927 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/53a770a64b47427c8d92058fdc13d2f6, entries=200, sequenceid=526, filesize=14.3 K 2023-03-27 04:56:15,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:15,935 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/f1913101cefc47df91521d2c94ecd23f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/f1913101cefc47df91521d2c94ecd23f 2023-03-27 04:56:15,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:15,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:15,935 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:15,935 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:15,935 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:15,935 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:15,935 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,935 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,936 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,936 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:15,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:15,936 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,936 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:15,941 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/f1913101cefc47df91521d2c94ecd23f, entries=150, sequenceid=526, filesize=11.9 K 2023-03-27 04:56:15,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:15,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:15,941 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:15,941 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:15,941 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:15,941 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:15,941 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:15,941 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:15,942 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:15,942 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:15,942 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:15,942 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:15,942 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:15,942 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,942 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:15,942 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:15,942 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/fbfbfa3826b84b73858bf4ff27ec74e1 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/fbfbfa3826b84b73858bf4ff27ec74e1 2023-03-27 04:56:15,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/fbfbfa3826b84b73858bf4ff27ec74e1, entries=150, sequenceid=526, filesize=11.9 K 2023-03-27 04:56:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,948 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,949 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,949 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,949 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e7756c618a32a862ad613327bee74d79 in 486ms, sequenceid=526, compaction requested=true 2023-03-27 04:56:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,949 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:15,950 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:15,950 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:15,950 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:15,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,950 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:15,950 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:15,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:15,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c1bc275[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:15,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:15,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,951 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:15,951 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:15,951 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/A is initiating minor compaction (all files) 2023-03-27 04:56:15,951 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/B is initiating minor compaction (all files) 2023-03-27 04:56:15,951 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/A in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:15,951 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/B in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:15,951 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/b0aa4815186c454ca2b6ebaea400505e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/767430fda6a546a2855522ecba91f372, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/53a770a64b47427c8d92058fdc13d2f6] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=39.1 K 2023-03-27 04:56:15,951 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/61514612cbf74eaa8830c0147e6d4ff8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/0166819a3fbb4d65a85baf2bfdbd4bc8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/f1913101cefc47df91521d2c94ecd23f] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=36.7 K 2023-03-27 04:56:15,951 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting b0aa4815186c454ca2b6ebaea400505e, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1679892974679 2023-03-27 04:56:15,952 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 61514612cbf74eaa8830c0147e6d4ff8, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1679892974679 2023-03-27 04:56:15,952 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 767430fda6a546a2855522ecba91f372, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1679892974827 2023-03-27 04:56:15,952 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 0166819a3fbb4d65a85baf2bfdbd4bc8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1679892974827 2023-03-27 04:56:15,952 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 53a770a64b47427c8d92058fdc13d2f6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1679892975362 2023-03-27 04:56:15,952 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting f1913101cefc47df91521d2c94ecd23f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1679892975365 2023-03-27 04:56:15,953 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,954 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,954 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:15,956 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:15,956 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:15,956 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:15,957 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:15,957 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:15,957 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,957 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:15,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:15,962 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:15,962 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:15,962 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:15,962 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:15,962 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:15,966 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#B#compaction#101 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:15,970 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,970 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:15,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:15,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:15,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,971 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,971 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:15,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:15,972 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#A#compaction#102 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,972 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:15,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:15,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:15,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:15,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:15,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:15,978 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:15,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:15,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:15,979 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,979 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:15,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:15,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:15,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:15,979 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:15,979 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:15,979 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:15,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:16,081 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:16,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:16,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:16,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:16,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:16,090 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:16,090 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:16,090 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:16,090 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:16,090 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:16,090 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:16,091 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:16,091 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:16,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:16,091 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,091 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:16,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:16,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:16,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:16,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:16,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:16,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:16,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:16,092 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:16,092 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:16,092 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:16,094 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:16,094 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:16,094 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:16,094 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:16,094 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-27 04:56:16,094 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:16,095 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:16,095 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:16,095 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:16,095 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:16,095 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:16,123 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/ed463fcc998a43a1bbdbe0019c3af252 2023-03-27 04:56:16,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:16,139 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:16,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:16,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49500 deadline: 1679893036146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:16,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893036147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:16,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893036149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:16,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49500 deadline: 1679893036250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:16,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893036253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:16,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893036255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,389 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/0f20dd5562334334940d338081e04f4d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/0f20dd5562334334940d338081e04f4d 2023-03-27 04:56:16,395 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/B of e7756c618a32a862ad613327bee74d79 into 0f20dd5562334334940d338081e04f4d(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:16,396 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:16,396 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/B, priority=13, startTime=1679892975950; duration=0sec 2023-03-27 04:56:16,396 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:16,396 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:16,398 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:16,399 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/C is initiating minor compaction (all files) 2023-03-27 04:56:16,399 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/C in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:16,399 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5c86f7d048864abd912b4353e01a5afc, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/b3209cb8a8464522bb6bbc2e62b51494, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/fbfbfa3826b84b73858bf4ff27ec74e1] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=36.7 K 2023-03-27 04:56:16,399 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/8161d81c3aa046b88ee1523f9cba9d4a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/8161d81c3aa046b88ee1523f9cba9d4a 2023-03-27 04:56:16,399 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 5c86f7d048864abd912b4353e01a5afc, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1679892974679 2023-03-27 04:56:16,402 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting b3209cb8a8464522bb6bbc2e62b51494, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1679892974827 2023-03-27 04:56:16,403 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting fbfbfa3826b84b73858bf4ff27ec74e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1679892975365 2023-03-27 04:56:16,410 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/A of e7756c618a32a862ad613327bee74d79 into 8161d81c3aa046b88ee1523f9cba9d4a(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:16,410 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:16,410 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/A, priority=13, startTime=1679892975949; duration=0sec 2023-03-27 04:56:16,411 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:16,420 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#C#compaction#105 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:16,447 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/3849187d767645a481fda2210d704f00 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/3849187d767645a481fda2210d704f00 2023-03-27 04:56:16,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:16,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49500 deadline: 1679893036454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:16,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893036457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,457 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e7756c618a32a862ad613327bee74d79/C of e7756c618a32a862ad613327bee74d79 into 3849187d767645a481fda2210d704f00(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:16,458 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:16,458 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/C, priority=13, startTime=1679892975950; duration=0sec 2023-03-27 04:56:16,458 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:16,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:16,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893036459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,537 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/3fc483882cd64eba8eb33dde2afeb9d7 2023-03-27 04:56:16,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:16,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49500 deadline: 1679893036759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:16,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893036759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:16,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893036761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,951 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/6d4ff319f2484a0a95b1f30a4cded47e 2023-03-27 04:56:16,956 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/ed463fcc998a43a1bbdbe0019c3af252 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/ed463fcc998a43a1bbdbe0019c3af252 2023-03-27 04:56:16,962 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/ed463fcc998a43a1bbdbe0019c3af252, entries=150, sequenceid=563, filesize=11.9 K 2023-03-27 04:56:16,963 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/3fc483882cd64eba8eb33dde2afeb9d7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/3fc483882cd64eba8eb33dde2afeb9d7 2023-03-27 04:56:16,977 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/3fc483882cd64eba8eb33dde2afeb9d7, entries=150, sequenceid=563, filesize=11.9 K 2023-03-27 04:56:16,978 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/6d4ff319f2484a0a95b1f30a4cded47e as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/6d4ff319f2484a0a95b1f30a4cded47e 2023-03-27 04:56:16,983 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/6d4ff319f2484a0a95b1f30a4cded47e, entries=150, sequenceid=563, filesize=11.9 K 2023-03-27 04:56:16,984 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for e7756c618a32a862ad613327bee74d79 in 889ms, sequenceid=563, compaction requested=false 2023-03-27 04:56:16,984 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:16,984 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:16,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:16,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:16,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:16,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:16,985 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,985 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:16,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:16,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:16,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:16,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:16,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:16,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:16,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:16,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:16,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@599a61d7[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:16,987 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:16,987 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:16,988 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:16,988 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,988 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:16,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:16,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:16,989 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:16,989 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:16,989 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:16,990 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:16,990 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:16,990 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,990 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:16,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:16,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:16,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:16,992 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:16,992 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:16,992 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:16,992 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:16,992 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:16,993 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:16,993 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:16,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:16,993 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:16,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:16,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:16,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:16,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:16,997 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:16,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:16,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,998 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:16,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:16,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:16,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:16,998 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:16,999 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:16,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:16,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:16,999 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:16,999 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:16,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:16,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:16,999 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,999 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:16,999 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:16,999 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:16,999 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:16,999 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:16,999 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:17,099 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:17,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:17,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:17,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:17,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:17,102 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:17,102 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:17,102 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:17,102 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:17,102 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:17,102 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:17,103 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:17,103 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:17,103 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:17,104 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:17,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:17,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:17,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:17,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:17,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:17,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:17,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:17,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:17,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:17,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:17,105 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:17,105 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:17,105 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-27 04:56:17,106 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:17,106 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:17,106 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:17,106 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:17,106 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:17,106 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:17,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:17,265 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:17,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:17,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893037284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:17,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679892977276, max timestamp=1679892977284Num uniques 859; 2023-03-27 04:56:17,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679892977276, max timestamp=1679892977284Num uniques 859; 2023-03-27 04:56:17,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:17,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679892977276, max timestamp=1679892977284Num uniques 859; 2023-03-27 04:56:17,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:17,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:17,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:17,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893037309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:17,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:17,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49500 deadline: 1679893037312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:17,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:17,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893037387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:17,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:17,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893037413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:17,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:17,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49500 deadline: 1679893037417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:17,516 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/c718ff1fdc554f81b9f3c16e97aa666e 2023-03-27 04:56:17,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:17,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893037591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:17,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:17,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893037617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:17,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:17,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49500 deadline: 1679893037622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:17,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:17,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49510 deadline: 1679893037781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:17,784 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8176 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., hostname=jenkins-hbase20.apache.org,44923,1679892955505, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:56:17,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:17,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893037898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:17,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:17,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893037923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:17,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:17,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49500 deadline: 1679893037929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:17,935 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/37b0b0b196644565923cf04b83b40ce4 2023-03-27 04:56:18,364 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/256a076fab0248c29a96dd4298a01603 2023-03-27 04:56:18,371 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/c718ff1fdc554f81b9f3c16e97aa666e as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c718ff1fdc554f81b9f3c16e97aa666e 2023-03-27 04:56:18,377 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c718ff1fdc554f81b9f3c16e97aa666e, entries=150, sequenceid=577, filesize=11.9 K 2023-03-27 04:56:18,382 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/37b0b0b196644565923cf04b83b40ce4 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/37b0b0b196644565923cf04b83b40ce4 2023-03-27 04:56:18,390 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/37b0b0b196644565923cf04b83b40ce4, entries=150, sequenceid=577, filesize=11.9 K 2023-03-27 04:56:18,391 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/256a076fab0248c29a96dd4298a01603 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/256a076fab0248c29a96dd4298a01603 2023-03-27 04:56:18,399 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/256a076fab0248c29a96dd4298a01603, entries=150, sequenceid=577, filesize=11.9 K 2023-03-27 04:56:18,400 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e7756c618a32a862ad613327bee74d79 in 1295ms, sequenceid=577, compaction requested=true 2023-03-27 04:56:18,400 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:18,400 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:18,400 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:18,400 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:18,400 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,400 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:18,400 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:18,402 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:18,402 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,403 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-27 04:56:18,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:18,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:18,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:18,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:18,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:18,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:18,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:18,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:18,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:18,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:18,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:18,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62272f96[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:18,405 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:18,405 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,406 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,406 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:18,406 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:18,406 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:18,406 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,406 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:18,407 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:18,407 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:18,407 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:18,407 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:18,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:18,409 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:18,409 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:18,409 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:18,409 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:18,410 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:18,412 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,412 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,412 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:18,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:18,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,414 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:18,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,421 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,423 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:18,423 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:18,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:18,423 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:18,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:18,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:18,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:18,423 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:18,423 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:18,424 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,424 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,424 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,424 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,424 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,424 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,424 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:18,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893038427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:18,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893038429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,437 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/75d6dabed8774d24b13d0aad3d749767 2023-03-27 04:56:18,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:18,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49500 deadline: 1679893038437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,523 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:18,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:18,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:18,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:18,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:18,527 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:18,527 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:18,527 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:18,527 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:18,530 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,530 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,531 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:18,531 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:18,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:18,531 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,531 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:18,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:18,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:18,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:18,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:18,533 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:18,533 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:18,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:18,533 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:18,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893038530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,533 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:18,533 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:18,533 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:18,533 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:18,534 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:18,534 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:18,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:18,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:18,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:18,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:18,535 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,536 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:18,536 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:18,536 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:18,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:18,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74fa5729[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:18,537 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:18,537 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,538 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,538 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:18,538 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:18,538 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:18,538 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,538 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:18,538 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,538 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:18,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:18,539 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:18,539 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:18,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:18,540 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:18,540 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:18,541 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:18,541 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:18,541 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:18,542 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:18,543 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:18,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,543 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:18,543 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:18,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,544 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,547 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,547 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,547 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:18,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:18,547 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:18,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:18,547 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:18,547 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:18,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:18,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:18,547 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,547 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,547 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,547 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,547 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:18,547 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:18,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:18,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:18,648 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:18,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:18,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:18,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:18,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:18,670 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:18,670 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:18,670 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:18,670 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:18,670 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,671 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,678 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:18,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,678 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,679 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,680 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:18,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:18,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:18,684 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:18,684 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:18,684 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:18,685 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:18,685 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:18,685 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:18,685 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:18,685 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:18,685 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:18,685 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:18,685 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:18,685 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,685 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:18,685 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:18,690 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,690 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:18,690 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,690 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:18,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:18,692 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:18,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56502ba9[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:18,692 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,695 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,695 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,695 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:18,695 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:18,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:18,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:18,697 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,697 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:18,697 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:18,697 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:18,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:18,708 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:18,708 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:18,708 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:18,708 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:18,708 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:18,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:18,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:18,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,714 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,715 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,723 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:18,723 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:18,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:18,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:18,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:18,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:18,723 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,724 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:18,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,724 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:18,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:18,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:18,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:18,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:18,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893038736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,824 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:18,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:18,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:18,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:18,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:18,827 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:18,827 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:18,827 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:18,827 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:18,827 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,827 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,828 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:18,828 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:18,828 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,828 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:18,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:18,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:18,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:18,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:18,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:18,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:18,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:18,829 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:18,829 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:18,829 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:18,829 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:18,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:18,830 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:18,830 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,830 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:18,830 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:18,830 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,830 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:18,831 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,831 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:18,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:18,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c4e6c6d[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:18,832 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:18,832 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,833 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,833 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:18,833 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:18,833 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,833 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:18,833 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:18,834 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,834 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:18,834 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:18,834 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:18,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:18,836 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:18,836 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:18,836 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:18,836 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:18,836 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:18,837 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,837 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:18,837 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:18,837 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,838 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:18,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,839 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:18,840 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:18,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:18,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:18,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:18,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:18,840 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:18,840 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:18,840 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:18,840 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:18,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:18,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/58143d7b94794ffda5afbe217fe77278 2023-03-27 04:56:18,884 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/ca456ab46bc04a078ea4242445de10d5 2023-03-27 04:56:18,890 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/75d6dabed8774d24b13d0aad3d749767 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/75d6dabed8774d24b13d0aad3d749767 2023-03-27 04:56:18,895 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/75d6dabed8774d24b13d0aad3d749767, entries=200, sequenceid=613, filesize=14.3 K 2023-03-27 04:56:18,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/58143d7b94794ffda5afbe217fe77278 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/58143d7b94794ffda5afbe217fe77278 2023-03-27 04:56:18,901 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/58143d7b94794ffda5afbe217fe77278, entries=150, sequenceid=613, filesize=11.9 K 2023-03-27 04:56:18,902 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/ca456ab46bc04a078ea4242445de10d5 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/ca456ab46bc04a078ea4242445de10d5 2023-03-27 04:56:18,912 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/ca456ab46bc04a078ea4242445de10d5, entries=150, sequenceid=613, filesize=11.9 K 2023-03-27 04:56:18,913 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for e7756c618a32a862ad613327bee74d79 in 511ms, sequenceid=613, compaction requested=true 2023-03-27 04:56:18,913 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:18,914 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:56:18,915 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52342 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:56:18,915 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/A is initiating minor compaction (all files) 2023-03-27 04:56:18,915 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/A in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:18,915 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/8161d81c3aa046b88ee1523f9cba9d4a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/ed463fcc998a43a1bbdbe0019c3af252, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c718ff1fdc554f81b9f3c16e97aa666e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/75d6dabed8774d24b13d0aad3d749767] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=51.1 K 2023-03-27 04:56:18,915 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 8161d81c3aa046b88ee1523f9cba9d4a, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1679892975365 2023-03-27 04:56:18,916 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:18,916 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting ed463fcc998a43a1bbdbe0019c3af252, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1679892975516 2023-03-27 04:56:18,916 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:56:18,916 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:18,916 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:18,916 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting c718ff1fdc554f81b9f3c16e97aa666e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1679892976141 2023-03-27 04:56:18,917 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 75d6dabed8774d24b13d0aad3d749767, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=613, earliestPutTs=1679892977306 2023-03-27 04:56:18,917 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49902 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:56:18,917 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/B is initiating minor compaction (all files) 2023-03-27 04:56:18,917 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/B in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:18,917 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/0f20dd5562334334940d338081e04f4d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/3fc483882cd64eba8eb33dde2afeb9d7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/37b0b0b196644565923cf04b83b40ce4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/58143d7b94794ffda5afbe217fe77278] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=48.7 K 2023-03-27 04:56:18,918 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 0f20dd5562334334940d338081e04f4d, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1679892975365 2023-03-27 04:56:18,918 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 3fc483882cd64eba8eb33dde2afeb9d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1679892975516 2023-03-27 04:56:18,919 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 37b0b0b196644565923cf04b83b40ce4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1679892976141 2023-03-27 04:56:18,919 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 58143d7b94794ffda5afbe217fe77278, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=613, earliestPutTs=1679892977308 2023-03-27 04:56:18,934 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#B#compaction#113 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:18,936 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#A#compaction#114 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:18,940 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:18,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:18,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:18,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:18,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:18,947 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:18,947 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:18,947 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:18,947 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:18,947 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,948 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:18,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:18,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:18,948 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,949 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:18,949 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:18,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:18,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:18,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:18,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:18,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:18,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:18,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:18,954 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:18,954 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:18,954 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:18,954 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:18,954 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:56:18,954 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:18,954 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:18,954 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:18,955 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:18,955 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:18,955 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:18,963 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/8845cc41983c4b1681d2c13c6afaed19 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/8845cc41983c4b1681d2c13c6afaed19 2023-03-27 04:56:18,970 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e7756c618a32a862ad613327bee74d79/A of e7756c618a32a862ad613327bee74d79 into 8845cc41983c4b1681d2c13c6afaed19(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:18,970 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:18,970 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/A, priority=12, startTime=1679892978914; duration=0sec 2023-03-27 04:56:18,970 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:18,970 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:56:18,971 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49868 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:56:18,971 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): e7756c618a32a862ad613327bee74d79/C is initiating minor compaction (all files) 2023-03-27 04:56:18,971 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e7756c618a32a862ad613327bee74d79/C in TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:18,972 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/3849187d767645a481fda2210d704f00, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/6d4ff319f2484a0a95b1f30a4cded47e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/256a076fab0248c29a96dd4298a01603, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/ca456ab46bc04a078ea4242445de10d5] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp, totalSize=48.7 K 2023-03-27 04:56:18,972 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 3849187d767645a481fda2210d704f00, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1679892975365 2023-03-27 04:56:18,973 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 6d4ff319f2484a0a95b1f30a4cded47e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1679892975516 2023-03-27 04:56:18,973 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/04cbdd9939694a329bb0b3fd75912962 2023-03-27 04:56:18,973 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 256a076fab0248c29a96dd4298a01603, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1679892976141 2023-03-27 04:56:18,974 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting ca456ab46bc04a078ea4242445de10d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=613, earliestPutTs=1679892977308 2023-03-27 04:56:18,981 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): e7756c618a32a862ad613327bee74d79#C#compaction#117 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:18,993 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/bd1803ca26c44b35bd4f3b97dca44f89 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bd1803ca26c44b35bd4f3b97dca44f89 2023-03-27 04:56:18,999 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e7756c618a32a862ad613327bee74d79/C of e7756c618a32a862ad613327bee74d79 into bd1803ca26c44b35bd4f3b97dca44f89(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:18,999 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:18,999 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/C, priority=12, startTime=1679892978916; duration=0sec 2023-03-27 04:56:18,999 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:19,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:19,044 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:19,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679892979069, max timestamp=1679892979074Num uniques 859; 2023-03-27 04:56:19,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679892979069, max timestamp=1679892979074Num uniques 859; 2023-03-27 04:56:19,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:19,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:19,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679892979069, max timestamp=1679892979074Num uniques 859; 2023-03-27 04:56:19,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:19,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:19,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893039125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:19,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893039229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,359 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/892adad167a34881bf4b241a23123673 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/892adad167a34881bf4b241a23123673 2023-03-27 04:56:19,365 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e7756c618a32a862ad613327bee74d79/B of e7756c618a32a862ad613327bee74d79 into 892adad167a34881bf4b241a23123673(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:19,365 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:19,365 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., storeName=e7756c618a32a862ad613327bee74d79/B, priority=12, startTime=1679892978916; duration=0sec 2023-03-27 04:56:19,365 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:19,385 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/7279d015135542c99b1d1defc87a3abf 2023-03-27 04:56:19,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:19,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49536 deadline: 1679893039389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,395 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79., hostname=jenkins-hbase20.apache.org,44923,1679892955505, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:56:19,418 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/6154e37b02954886bad4334892faa2a7 2023-03-27 04:56:19,433 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/04cbdd9939694a329bb0b3fd75912962 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/04cbdd9939694a329bb0b3fd75912962 2023-03-27 04:56:19,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:19,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49552 deadline: 1679893039435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:19,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49524 deadline: 1679893039438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,442 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/04cbdd9939694a329bb0b3fd75912962, entries=150, sequenceid=623, filesize=11.9 K 2023-03-27 04:56:19,446 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-27 04:56:19,446 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-27 04:56:19,447 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/7279d015135542c99b1d1defc87a3abf as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/7279d015135542c99b1d1defc87a3abf 2023-03-27 04:56:19,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:19,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49500 deadline: 1679893039448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,454 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/7279d015135542c99b1d1defc87a3abf, entries=150, sequenceid=623, filesize=11.9 K 2023-03-27 04:56:19,455 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/6154e37b02954886bad4334892faa2a7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/6154e37b02954886bad4334892faa2a7 2023-03-27 04:56:19,467 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/6154e37b02954886bad4334892faa2a7, entries=150, sequenceid=623, filesize=11.9 K 2023-03-27 04:56:19,469 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e7756c618a32a862ad613327bee74d79 in 515ms, sequenceid=623, compaction requested=false 2023-03-27 04:56:19,469 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:19,469 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:19,469 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:19,469 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:19,469 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,469 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:19,469 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:19,470 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:19,470 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,470 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:19,470 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:19,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:19,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:19,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:19,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:19,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74eb3ed[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:19,479 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:19,479 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:19,497 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:19,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:19,497 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:19,497 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:19,497 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:19,501 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,501 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:19,501 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:19,502 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:19,502 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:19,502 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:19,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:19,505 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:19,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:19,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:19,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:19,506 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:19,507 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:19,507 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:19,507 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:19,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:19,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:19,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:19,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:19,509 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:19,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:19,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:19,510 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,511 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,511 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:19,511 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:19,511 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,511 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:19,511 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:19,511 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:19,511 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:19,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:19,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:19,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:19,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:19,511 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:19,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:19,512 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:19,512 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:19,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:19,612 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:19,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:19,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:19,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:19,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:19,615 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:19,615 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:19,616 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:19,616 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:19,616 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:19,616 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:19,617 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:19,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:19,617 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:19,617 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:19,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:19,617 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:19,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:19,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:19,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:19,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:19,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:19,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:19,618 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:19,618 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:19,618 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:19,618 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:19,618 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:19,618 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:19,618 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. started... 2023-03-27 04:56:19,619 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-27 04:56:19,619 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:19,619 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:19,619 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:19,619 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:19,619 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:19,619 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:19,711 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3fed686f to 127.0.0.1:61059 2023-03-27 04:56:19,712 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:19,712 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a7ef79d to 127.0.0.1:61059 2023-03-27 04:56:19,712 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:19,712 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1439cba9 to 127.0.0.1:61059 2023-03-27 04:56:19,713 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:19,720 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ab9467e to 127.0.0.1:61059 2023-03-27 04:56:19,720 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:19,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:19,739 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. as already flushing 2023-03-27 04:56:19,739 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4654132d to 127.0.0.1:61059 2023-03-27 04:56:19,739 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:20,031 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/77572504a05f4242bdad6f9fbd03c7dd 2023-03-27 04:56:20,444 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/2c81c5e80c714eec98d747a831d91766 2023-03-27 04:56:20,857 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/381ffcd3143c4828b93ce1459c562eff 2023-03-27 04:56:20,864 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/77572504a05f4242bdad6f9fbd03c7dd as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/77572504a05f4242bdad6f9fbd03c7dd 2023-03-27 04:56:20,869 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/77572504a05f4242bdad6f9fbd03c7dd, entries=150, sequenceid=663, filesize=11.9 K 2023-03-27 04:56:20,870 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/2c81c5e80c714eec98d747a831d91766 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/2c81c5e80c714eec98d747a831d91766 2023-03-27 04:56:20,874 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/2c81c5e80c714eec98d747a831d91766, entries=150, sequenceid=663, filesize=11.9 K 2023-03-27 04:56:20,875 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/381ffcd3143c4828b93ce1459c562eff as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/381ffcd3143c4828b93ce1459c562eff 2023-03-27 04:56:20,880 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/381ffcd3143c4828b93ce1459c562eff, entries=150, sequenceid=663, filesize=11.9 K 2023-03-27 04:56:20,880 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=6.71 KB/6870 for e7756c618a32a862ad613327bee74d79 in 1262ms, sequenceid=663, compaction requested=true 2023-03-27 04:56:20,881 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:20,881 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:20,881 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:20,881 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:20,881 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:20,881 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:20,881 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:20,882 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:20,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:20,883 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:20,883 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:20,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:20,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@212bff32[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:20,884 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:20,884 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:20,885 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:20,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:20,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:20,885 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:20,885 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:20,885 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:20,885 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:20,885 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:20,886 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:20,886 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:20,886 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:20,886 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:20,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:20,888 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:20,888 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:20,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:20,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:20,889 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:20,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:20,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:20,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:20,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:20,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:20,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:20,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:20,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:20,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:20,901 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:20,902 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:20,906 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:20,906 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:20,906 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:20,907 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:20,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:20,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:20,906 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:20,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:20,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:20,907 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:20,907 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:20,907 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:20,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:20,907 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:20,907 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:20,907 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:20,907 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:21,007 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:21,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:21,459 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00e18ba3 to 127.0.0.1:61059 2023-03-27 04:56:21,460 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:21,465 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e0ba002 to 127.0.0.1:61059 2023-03-27 04:56:21,465 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:27,837 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x136956fd to 127.0.0.1:61059 2023-03-27 04:56:27,838 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:29,487 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77443fb0 to 127.0.0.1:61059 2023-03-27 04:56:29,487 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:29,487 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-27 04:56:29,488 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 88 2023-03-27 04:56:29,488 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 81 2023-03-27 04:56:29,488 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 74 2023-03-27 04:56:29,488 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 203 2023-03-27 04:56:29,489 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 98 2023-03-27 04:56:29,489 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-27 04:56:29,489 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(398): read 4041 2023-03-27 04:56:29,489 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(398): read 3836 2023-03-27 04:56:29,489 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-27 04:56:29,489 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(402): scanned 1783 2023-03-27 04:56:29,489 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(403): verified 5343 rows 2023-03-27 04:56:29,489 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(402): scanned 1779 2023-03-27 04:56:29,490 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(403): verified 5337 rows 2023-03-27 04:56:29,490 INFO [Listener at localhost.localdomain/43311] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-27 04:56:29,490 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12582a26 to 127.0.0.1:61059 2023-03-27 04:56:29,490 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:29,497 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-27 04:56:29,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-27 04:56:29,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-27 04:56:29,509 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679892989509"}]},"ts":"1679892989509"} 2023-03-27 04:56:29,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-27 04:56:29,510 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-27 04:56:29,519 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-27 04:56:29,521 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e7756c618a32a862ad613327bee74d79, UNASSIGN}] 2023-03-27 04:56:29,523 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e7756c618a32a862ad613327bee74d79, UNASSIGN 2023-03-27 04:56:29,524 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=e7756c618a32a862ad613327bee74d79, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:29,524 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679892989524"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679892989524"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679892989524"}]},"ts":"1679892989524"} 2023-03-27 04:56:29,525 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:56:29,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-27 04:56:29,688 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:29,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing e7756c618a32a862ad613327bee74d79, disabling compactions & flushes 2023-03-27 04:56:29,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:29,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:29,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. after waiting 0 ms 2023-03-27 04:56:29,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:29,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing e7756c618a32a862ad613327bee74d79 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-27 04:56:29,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=A 2023-03-27 04:56:29,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:29,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=B 2023-03-27 04:56:29,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:29,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e7756c618a32a862ad613327bee74d79, store=C 2023-03-27 04:56:29,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:29,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-27 04:56:30,100 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=671 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/fa374c80d0b0492c97861edbc118f435 2023-03-27 04:56:30,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-27 04:56:30,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=671 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/a57b6e06947149b78d08b81eb56535bd 2023-03-27 04:56:30,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-27 04:56:30,936 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=671 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/f0529fed43a24f8c88ce5b439115ce5c 2023-03-27 04:56:30,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/A/fa374c80d0b0492c97861edbc118f435 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/fa374c80d0b0492c97861edbc118f435 2023-03-27 04:56:30,950 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/fa374c80d0b0492c97861edbc118f435, entries=150, sequenceid=671, filesize=11.9 K 2023-03-27 04:56:30,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/B/a57b6e06947149b78d08b81eb56535bd as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/a57b6e06947149b78d08b81eb56535bd 2023-03-27 04:56:30,955 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/a57b6e06947149b78d08b81eb56535bd, entries=150, sequenceid=671, filesize=11.9 K 2023-03-27 04:56:30,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/.tmp/C/f0529fed43a24f8c88ce5b439115ce5c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f0529fed43a24f8c88ce5b439115ce5c 2023-03-27 04:56:30,960 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f0529fed43a24f8c88ce5b439115ce5c, entries=150, sequenceid=671, filesize=11.9 K 2023-03-27 04:56:30,961 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for e7756c618a32a862ad613327bee74d79 in 1271ms, sequenceid=671, compaction requested=true 2023-03-27 04:56:30,979 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/140dc831fe7440d4aa0471f4933bbd28, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/26c637eefad242bca77883e69e6da2a7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/662fc6e3873141ce9b973e562a247c73, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/f3f9256101324bab8d6b02400c53bf2f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c0247ebceddb41e1b04f2bb3fd343dc8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d486958c13dc47829f9475717b8ddb82, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c8fc1940f8554d838d5b79ebd5127c43, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/f3adbfa199c549f4a2b893331fb554a9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d914c443343745ab8b2f769120fe59b0, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/ac325438900d406f81ba8d8dca300921, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/93994c938ae946bcaa436935e0285573, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/7b7d682992424da5a445310459d5ae50, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/1d5e7284f91047c4a9ef37140c7031d9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/cbd9112850234648ad56b305389b186f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c10aeccba94645e29670b453be1207e9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/2563e834e7394560b8cf8cc3529e9721, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/73c035ebac834991a52dd29bc11502f7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d155a5e080fb47aa90ded1b62940711a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/9cdd728b094e42d781c7c0186ad26de2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/99fee2c7e86e4607b465b6e67ff9d35c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/b64a3a320207416ba8cc68af0323f875, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/97298b673b1a497ca3c72f6746494f5d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/f73c4a19c43c49149dba506d5ada5438, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/8be95bcbcb75465a9483d5b8c4f15e0b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/29cbf7aa65164963b6f229b1c2bcb6d6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/9013b51b608344dfb98322c79293f639, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/dc6d14ef1cdc406dbd5c8ad91d7dedd5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/51100602df744299ac78f0bac5740855, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/7088d1cdc62e4710b414e88a1ab62bf2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/e02fc3ca85124b8cb4f20029d003a41c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/12332f5d09cc4333afa6ea574360a1b4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/b0aa4815186c454ca2b6ebaea400505e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/767430fda6a546a2855522ecba91f372, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/53a770a64b47427c8d92058fdc13d2f6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/8161d81c3aa046b88ee1523f9cba9d4a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/ed463fcc998a43a1bbdbe0019c3af252, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c718ff1fdc554f81b9f3c16e97aa666e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/75d6dabed8774d24b13d0aad3d749767] to archive 2023-03-27 04:56:30,981 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:56:30,986 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/140dc831fe7440d4aa0471f4933bbd28 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/140dc831fe7440d4aa0471f4933bbd28 2023-03-27 04:56:30,987 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/26c637eefad242bca77883e69e6da2a7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/26c637eefad242bca77883e69e6da2a7 2023-03-27 04:56:30,989 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/662fc6e3873141ce9b973e562a247c73 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/662fc6e3873141ce9b973e562a247c73 2023-03-27 04:56:30,990 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/f3f9256101324bab8d6b02400c53bf2f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/f3f9256101324bab8d6b02400c53bf2f 2023-03-27 04:56:30,991 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c0247ebceddb41e1b04f2bb3fd343dc8 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c0247ebceddb41e1b04f2bb3fd343dc8 2023-03-27 04:56:30,992 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d486958c13dc47829f9475717b8ddb82 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d486958c13dc47829f9475717b8ddb82 2023-03-27 04:56:30,993 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c8fc1940f8554d838d5b79ebd5127c43 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c8fc1940f8554d838d5b79ebd5127c43 2023-03-27 04:56:30,995 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/f3adbfa199c549f4a2b893331fb554a9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/f3adbfa199c549f4a2b893331fb554a9 2023-03-27 04:56:30,996 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d914c443343745ab8b2f769120fe59b0 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d914c443343745ab8b2f769120fe59b0 2023-03-27 04:56:30,997 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/ac325438900d406f81ba8d8dca300921 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/ac325438900d406f81ba8d8dca300921 2023-03-27 04:56:30,998 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/93994c938ae946bcaa436935e0285573 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/93994c938ae946bcaa436935e0285573 2023-03-27 04:56:30,999 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/7b7d682992424da5a445310459d5ae50 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/7b7d682992424da5a445310459d5ae50 2023-03-27 04:56:31,000 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/1d5e7284f91047c4a9ef37140c7031d9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/1d5e7284f91047c4a9ef37140c7031d9 2023-03-27 04:56:31,001 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/cbd9112850234648ad56b305389b186f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/cbd9112850234648ad56b305389b186f 2023-03-27 04:56:31,004 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c10aeccba94645e29670b453be1207e9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c10aeccba94645e29670b453be1207e9 2023-03-27 04:56:31,006 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/2563e834e7394560b8cf8cc3529e9721 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/2563e834e7394560b8cf8cc3529e9721 2023-03-27 04:56:31,008 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/73c035ebac834991a52dd29bc11502f7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/73c035ebac834991a52dd29bc11502f7 2023-03-27 04:56:31,009 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d155a5e080fb47aa90ded1b62940711a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/d155a5e080fb47aa90ded1b62940711a 2023-03-27 04:56:31,011 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/9cdd728b094e42d781c7c0186ad26de2 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/9cdd728b094e42d781c7c0186ad26de2 2023-03-27 04:56:31,012 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/99fee2c7e86e4607b465b6e67ff9d35c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/99fee2c7e86e4607b465b6e67ff9d35c 2023-03-27 04:56:31,013 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/b64a3a320207416ba8cc68af0323f875 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/b64a3a320207416ba8cc68af0323f875 2023-03-27 04:56:31,014 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/97298b673b1a497ca3c72f6746494f5d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/97298b673b1a497ca3c72f6746494f5d 2023-03-27 04:56:31,015 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/f73c4a19c43c49149dba506d5ada5438 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/f73c4a19c43c49149dba506d5ada5438 2023-03-27 04:56:31,017 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/8be95bcbcb75465a9483d5b8c4f15e0b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/8be95bcbcb75465a9483d5b8c4f15e0b 2023-03-27 04:56:31,018 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/29cbf7aa65164963b6f229b1c2bcb6d6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/29cbf7aa65164963b6f229b1c2bcb6d6 2023-03-27 04:56:31,019 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/9013b51b608344dfb98322c79293f639 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/9013b51b608344dfb98322c79293f639 2023-03-27 04:56:31,020 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/dc6d14ef1cdc406dbd5c8ad91d7dedd5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/dc6d14ef1cdc406dbd5c8ad91d7dedd5 2023-03-27 04:56:31,021 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/51100602df744299ac78f0bac5740855 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/51100602df744299ac78f0bac5740855 2023-03-27 04:56:31,022 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/7088d1cdc62e4710b414e88a1ab62bf2 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/7088d1cdc62e4710b414e88a1ab62bf2 2023-03-27 04:56:31,023 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/e02fc3ca85124b8cb4f20029d003a41c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/e02fc3ca85124b8cb4f20029d003a41c 2023-03-27 04:56:31,024 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/12332f5d09cc4333afa6ea574360a1b4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/12332f5d09cc4333afa6ea574360a1b4 2023-03-27 04:56:31,025 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/b0aa4815186c454ca2b6ebaea400505e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/b0aa4815186c454ca2b6ebaea400505e 2023-03-27 04:56:31,026 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/767430fda6a546a2855522ecba91f372 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/767430fda6a546a2855522ecba91f372 2023-03-27 04:56:31,028 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/53a770a64b47427c8d92058fdc13d2f6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/53a770a64b47427c8d92058fdc13d2f6 2023-03-27 04:56:31,029 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/8161d81c3aa046b88ee1523f9cba9d4a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/8161d81c3aa046b88ee1523f9cba9d4a 2023-03-27 04:56:31,030 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/ed463fcc998a43a1bbdbe0019c3af252 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/ed463fcc998a43a1bbdbe0019c3af252 2023-03-27 04:56:31,032 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c718ff1fdc554f81b9f3c16e97aa666e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/c718ff1fdc554f81b9f3c16e97aa666e 2023-03-27 04:56:31,033 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/75d6dabed8774d24b13d0aad3d749767 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/75d6dabed8774d24b13d0aad3d749767 2023-03-27 04:56:31,055 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/6ac434dd19104755959a879710ce83e7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/935ce82e8afc403c82c7d4afa20b7151, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/ab847b59941747fdbdf60f14f1e92094, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/ccf3f27ea5144588bbaad9db8cf0f76b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/e06010bd47234d3fa6153d310d495cf9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/461e6e18baad45cab26d014a151b1f3f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/cb431bf289ce4262ae1bd815e6d013d5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/11f723e8a88a4d3f9256d358ac156ba8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/9aa15ebb97f044dab136a04fb6eab2e1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/a4b82fe557e344ddab2f55a6257f97ef, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/646aa30c434441589179b9d3f3c68535, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/633f445a5f6a4641ad95ecdae1bacb11, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/8903952b934d4d5d89fe6a8151d60b02, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/bacbb614450b4b948b490b67950804e8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/25708bc12dea4ec58493998793728f16, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/b16cf732665f44c8a1ebb36df7d5f845, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/57a46d0439bf4c9aa3ca358aee49eb02, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/0a8888ea92ef4150975bdf366ab492c5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/34ab99085ceb4d74a9d04b1dc7d28862, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/a845b5eea35748a08700dc3b71af75b9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/fc1ef4bb68e1428a943a43f06fb879b8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/e3cf67a406904dcdb5fb18419d67df62, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/16511e8dbbb74283a1e80a6e6aef5051, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/6c61cc3dd7b24f6eb4f27481674f1d79, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/dc13754af7a44ab48d65af23d927de10, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/99944657dce2447ea56e9bbd3104bb44, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/31773a73379f41538f83453d62ab8926, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/6f3b43f27a3f4a429f027668ff52e5ab, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/137f6d525970442aa8d94fa40c5851e4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/aaa5bac5e49d430a95a124aa6e58fc77, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/61514612cbf74eaa8830c0147e6d4ff8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/5854badf28184cfa877f6198ea4da89a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/0166819a3fbb4d65a85baf2bfdbd4bc8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/0f20dd5562334334940d338081e04f4d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/f1913101cefc47df91521d2c94ecd23f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/3fc483882cd64eba8eb33dde2afeb9d7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/37b0b0b196644565923cf04b83b40ce4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/58143d7b94794ffda5afbe217fe77278] to archive 2023-03-27 04:56:31,057 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:56:31,059 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/6ac434dd19104755959a879710ce83e7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/6ac434dd19104755959a879710ce83e7 2023-03-27 04:56:31,060 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/935ce82e8afc403c82c7d4afa20b7151 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/935ce82e8afc403c82c7d4afa20b7151 2023-03-27 04:56:31,061 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/ab847b59941747fdbdf60f14f1e92094 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/ab847b59941747fdbdf60f14f1e92094 2023-03-27 04:56:31,063 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/ccf3f27ea5144588bbaad9db8cf0f76b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/ccf3f27ea5144588bbaad9db8cf0f76b 2023-03-27 04:56:31,064 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/e06010bd47234d3fa6153d310d495cf9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/e06010bd47234d3fa6153d310d495cf9 2023-03-27 04:56:31,065 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/461e6e18baad45cab26d014a151b1f3f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/461e6e18baad45cab26d014a151b1f3f 2023-03-27 04:56:31,066 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/cb431bf289ce4262ae1bd815e6d013d5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/cb431bf289ce4262ae1bd815e6d013d5 2023-03-27 04:56:31,067 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/11f723e8a88a4d3f9256d358ac156ba8 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/11f723e8a88a4d3f9256d358ac156ba8 2023-03-27 04:56:31,068 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/9aa15ebb97f044dab136a04fb6eab2e1 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/9aa15ebb97f044dab136a04fb6eab2e1 2023-03-27 04:56:31,069 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/a4b82fe557e344ddab2f55a6257f97ef to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/a4b82fe557e344ddab2f55a6257f97ef 2023-03-27 04:56:31,070 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/646aa30c434441589179b9d3f3c68535 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/646aa30c434441589179b9d3f3c68535 2023-03-27 04:56:31,071 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/633f445a5f6a4641ad95ecdae1bacb11 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/633f445a5f6a4641ad95ecdae1bacb11 2023-03-27 04:56:31,072 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/8903952b934d4d5d89fe6a8151d60b02 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/8903952b934d4d5d89fe6a8151d60b02 2023-03-27 04:56:31,073 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/bacbb614450b4b948b490b67950804e8 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/bacbb614450b4b948b490b67950804e8 2023-03-27 04:56:31,074 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/25708bc12dea4ec58493998793728f16 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/25708bc12dea4ec58493998793728f16 2023-03-27 04:56:31,075 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/b16cf732665f44c8a1ebb36df7d5f845 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/b16cf732665f44c8a1ebb36df7d5f845 2023-03-27 04:56:31,076 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/57a46d0439bf4c9aa3ca358aee49eb02 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/57a46d0439bf4c9aa3ca358aee49eb02 2023-03-27 04:56:31,077 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/0a8888ea92ef4150975bdf366ab492c5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/0a8888ea92ef4150975bdf366ab492c5 2023-03-27 04:56:31,078 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/34ab99085ceb4d74a9d04b1dc7d28862 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/34ab99085ceb4d74a9d04b1dc7d28862 2023-03-27 04:56:31,079 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/a845b5eea35748a08700dc3b71af75b9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/a845b5eea35748a08700dc3b71af75b9 2023-03-27 04:56:31,080 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/fc1ef4bb68e1428a943a43f06fb879b8 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/fc1ef4bb68e1428a943a43f06fb879b8 2023-03-27 04:56:31,081 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/e3cf67a406904dcdb5fb18419d67df62 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/e3cf67a406904dcdb5fb18419d67df62 2023-03-27 04:56:31,082 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/16511e8dbbb74283a1e80a6e6aef5051 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/16511e8dbbb74283a1e80a6e6aef5051 2023-03-27 04:56:31,083 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/6c61cc3dd7b24f6eb4f27481674f1d79 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/6c61cc3dd7b24f6eb4f27481674f1d79 2023-03-27 04:56:31,084 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/dc13754af7a44ab48d65af23d927de10 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/dc13754af7a44ab48d65af23d927de10 2023-03-27 04:56:31,085 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/99944657dce2447ea56e9bbd3104bb44 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/99944657dce2447ea56e9bbd3104bb44 2023-03-27 04:56:31,086 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/31773a73379f41538f83453d62ab8926 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/31773a73379f41538f83453d62ab8926 2023-03-27 04:56:31,087 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/6f3b43f27a3f4a429f027668ff52e5ab to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/6f3b43f27a3f4a429f027668ff52e5ab 2023-03-27 04:56:31,088 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/137f6d525970442aa8d94fa40c5851e4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/137f6d525970442aa8d94fa40c5851e4 2023-03-27 04:56:31,089 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/aaa5bac5e49d430a95a124aa6e58fc77 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/aaa5bac5e49d430a95a124aa6e58fc77 2023-03-27 04:56:31,090 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/61514612cbf74eaa8830c0147e6d4ff8 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/61514612cbf74eaa8830c0147e6d4ff8 2023-03-27 04:56:31,091 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/5854badf28184cfa877f6198ea4da89a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/5854badf28184cfa877f6198ea4da89a 2023-03-27 04:56:31,092 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/0166819a3fbb4d65a85baf2bfdbd4bc8 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/0166819a3fbb4d65a85baf2bfdbd4bc8 2023-03-27 04:56:31,093 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/0f20dd5562334334940d338081e04f4d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/0f20dd5562334334940d338081e04f4d 2023-03-27 04:56:31,094 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/f1913101cefc47df91521d2c94ecd23f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/f1913101cefc47df91521d2c94ecd23f 2023-03-27 04:56:31,095 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/3fc483882cd64eba8eb33dde2afeb9d7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/3fc483882cd64eba8eb33dde2afeb9d7 2023-03-27 04:56:31,096 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/37b0b0b196644565923cf04b83b40ce4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/37b0b0b196644565923cf04b83b40ce4 2023-03-27 04:56:31,097 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/58143d7b94794ffda5afbe217fe77278 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/58143d7b94794ffda5afbe217fe77278 2023-03-27 04:56:31,102 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bfa559db3dce45169e963f3c5cfbdca8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bbef6359fbd94d989716595a3370ba35, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/52b6b6ff5fef471692c67d51fadc60dd, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/a107ec6aeaf64992adb1e32a594c0a44, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f8b3d0d0cf344ca796f4df837b0ecce0, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/4665b9efd869472498f9c6a7dc2ad8c2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/e01a3ce1cb584df3898c04eb8a3e8735, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/698b505a609d4bbda96ace1a22f880c2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/16ded63a4ec746d985aa1ad4f2094bfb, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5659f41a17d04a3a8a8e57f001d87d3d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/aabf3372470941ba9cd3ae716878338d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/02214ebea5bd43a3a88a0559360a3f61, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/48424308f6984b7eb29226b4a008d20c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5ea2f681a5824941919d3f3ad82e7693, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bdc3f5acfe394173b0b919de7daa4003, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/3a91f7aabb094ae68185c2cfcb448f5d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/6ac5e06a05854070bfce4feba1b282a4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/fea5d17b9cf84edbad27556e8bee3434, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/15ccc8f8855e4df3957e0f54f37b1154, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/4ff6928b52a74108943ca02eed9dca48, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f1176b8cc9644ad1952ca83d2dc06900, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/abd87f04c5f64272811114046175d748, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f49ec8bb4469471a94e7298c707554a7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/20a36c5ba87a4ffba6f258f84f83f4fa, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/72b96b45337d4b93966aaf1ec315320d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5c8d45ccb1f64729a4caca3f298ae098, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/56312b5b0a7f4acc87dbc3d74377ec5c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/c4d6a92436c0435d88b5a81b4b6a2529, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/548a54be9b2a457e8134e8c6e5c4322d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5c86f7d048864abd912b4353e01a5afc, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/7e17ccd1844745218f5ebb28a639ad69, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/b3209cb8a8464522bb6bbc2e62b51494, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/3849187d767645a481fda2210d704f00, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/fbfbfa3826b84b73858bf4ff27ec74e1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/6d4ff319f2484a0a95b1f30a4cded47e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/256a076fab0248c29a96dd4298a01603, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/ca456ab46bc04a078ea4242445de10d5] to archive 2023-03-27 04:56:31,103 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:56:31,105 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bfa559db3dce45169e963f3c5cfbdca8 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bfa559db3dce45169e963f3c5cfbdca8 2023-03-27 04:56:31,106 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bbef6359fbd94d989716595a3370ba35 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bbef6359fbd94d989716595a3370ba35 2023-03-27 04:56:31,108 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/52b6b6ff5fef471692c67d51fadc60dd to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/52b6b6ff5fef471692c67d51fadc60dd 2023-03-27 04:56:31,109 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/a107ec6aeaf64992adb1e32a594c0a44 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/a107ec6aeaf64992adb1e32a594c0a44 2023-03-27 04:56:31,110 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f8b3d0d0cf344ca796f4df837b0ecce0 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f8b3d0d0cf344ca796f4df837b0ecce0 2023-03-27 04:56:31,111 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/4665b9efd869472498f9c6a7dc2ad8c2 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/4665b9efd869472498f9c6a7dc2ad8c2 2023-03-27 04:56:31,113 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/e01a3ce1cb584df3898c04eb8a3e8735 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/e01a3ce1cb584df3898c04eb8a3e8735 2023-03-27 04:56:31,114 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/698b505a609d4bbda96ace1a22f880c2 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/698b505a609d4bbda96ace1a22f880c2 2023-03-27 04:56:31,115 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/16ded63a4ec746d985aa1ad4f2094bfb to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/16ded63a4ec746d985aa1ad4f2094bfb 2023-03-27 04:56:31,116 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5659f41a17d04a3a8a8e57f001d87d3d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5659f41a17d04a3a8a8e57f001d87d3d 2023-03-27 04:56:31,118 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/aabf3372470941ba9cd3ae716878338d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/aabf3372470941ba9cd3ae716878338d 2023-03-27 04:56:31,119 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/02214ebea5bd43a3a88a0559360a3f61 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/02214ebea5bd43a3a88a0559360a3f61 2023-03-27 04:56:31,120 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/48424308f6984b7eb29226b4a008d20c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/48424308f6984b7eb29226b4a008d20c 2023-03-27 04:56:31,121 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5ea2f681a5824941919d3f3ad82e7693 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5ea2f681a5824941919d3f3ad82e7693 2023-03-27 04:56:31,123 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bdc3f5acfe394173b0b919de7daa4003 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bdc3f5acfe394173b0b919de7daa4003 2023-03-27 04:56:31,125 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/3a91f7aabb094ae68185c2cfcb448f5d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/3a91f7aabb094ae68185c2cfcb448f5d 2023-03-27 04:56:31,127 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/6ac5e06a05854070bfce4feba1b282a4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/6ac5e06a05854070bfce4feba1b282a4 2023-03-27 04:56:31,128 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/fea5d17b9cf84edbad27556e8bee3434 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/fea5d17b9cf84edbad27556e8bee3434 2023-03-27 04:56:31,129 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/15ccc8f8855e4df3957e0f54f37b1154 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/15ccc8f8855e4df3957e0f54f37b1154 2023-03-27 04:56:31,130 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/4ff6928b52a74108943ca02eed9dca48 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/4ff6928b52a74108943ca02eed9dca48 2023-03-27 04:56:31,131 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f1176b8cc9644ad1952ca83d2dc06900 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f1176b8cc9644ad1952ca83d2dc06900 2023-03-27 04:56:31,132 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/abd87f04c5f64272811114046175d748 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/abd87f04c5f64272811114046175d748 2023-03-27 04:56:31,133 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f49ec8bb4469471a94e7298c707554a7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f49ec8bb4469471a94e7298c707554a7 2023-03-27 04:56:31,134 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/20a36c5ba87a4ffba6f258f84f83f4fa to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/20a36c5ba87a4ffba6f258f84f83f4fa 2023-03-27 04:56:31,135 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/72b96b45337d4b93966aaf1ec315320d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/72b96b45337d4b93966aaf1ec315320d 2023-03-27 04:56:31,136 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5c8d45ccb1f64729a4caca3f298ae098 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5c8d45ccb1f64729a4caca3f298ae098 2023-03-27 04:56:31,136 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/56312b5b0a7f4acc87dbc3d74377ec5c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/56312b5b0a7f4acc87dbc3d74377ec5c 2023-03-27 04:56:31,137 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/c4d6a92436c0435d88b5a81b4b6a2529 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/c4d6a92436c0435d88b5a81b4b6a2529 2023-03-27 04:56:31,139 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/548a54be9b2a457e8134e8c6e5c4322d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/548a54be9b2a457e8134e8c6e5c4322d 2023-03-27 04:56:31,140 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5c86f7d048864abd912b4353e01a5afc to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/5c86f7d048864abd912b4353e01a5afc 2023-03-27 04:56:31,141 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/7e17ccd1844745218f5ebb28a639ad69 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/7e17ccd1844745218f5ebb28a639ad69 2023-03-27 04:56:31,142 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/b3209cb8a8464522bb6bbc2e62b51494 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/b3209cb8a8464522bb6bbc2e62b51494 2023-03-27 04:56:31,143 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/3849187d767645a481fda2210d704f00 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/3849187d767645a481fda2210d704f00 2023-03-27 04:56:31,144 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/fbfbfa3826b84b73858bf4ff27ec74e1 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/fbfbfa3826b84b73858bf4ff27ec74e1 2023-03-27 04:56:31,145 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/6d4ff319f2484a0a95b1f30a4cded47e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/6d4ff319f2484a0a95b1f30a4cded47e 2023-03-27 04:56:31,146 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/256a076fab0248c29a96dd4298a01603 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/256a076fab0248c29a96dd4298a01603 2023-03-27 04:56:31,147 DEBUG [StoreCloser-TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/ca456ab46bc04a078ea4242445de10d5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/ca456ab46bc04a078ea4242445de10d5 2023-03-27 04:56:31,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/recovered.edits/674.seqid, newMaxSeqId=674, maxSeqId=1 2023-03-27 04:56:31,153 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79. 2023-03-27 04:56:31,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for e7756c618a32a862ad613327bee74d79: 2023-03-27 04:56:31,154 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:31,155 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=e7756c618a32a862ad613327bee74d79, regionState=CLOSED 2023-03-27 04:56:31,155 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679892991155"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679892991155"}]},"ts":"1679892991155"} 2023-03-27 04:56:31,157 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-27 04:56:31,158 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure e7756c618a32a862ad613327bee74d79, server=jenkins-hbase20.apache.org,44923,1679892955505 in 1.6310 sec 2023-03-27 04:56:31,159 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-27 04:56:31,159 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e7756c618a32a862ad613327bee74d79, UNASSIGN in 1.6360 sec 2023-03-27 04:56:31,159 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679892991159"}]},"ts":"1679892991159"} 2023-03-27 04:56:31,160 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-27 04:56:31,161 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-27 04:56:31,163 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6600 sec 2023-03-27 04:56:31,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-27 04:56:31,618 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-27 04:56:31,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-27 04:56:31,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:56:31,633 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:56:31,635 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:56:31,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-27 04:56:31,640 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:31,644 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/recovered.edits] 2023-03-27 04:56:31,648 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/04cbdd9939694a329bb0b3fd75912962 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/04cbdd9939694a329bb0b3fd75912962 2023-03-27 04:56:31,650 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/77572504a05f4242bdad6f9fbd03c7dd to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/77572504a05f4242bdad6f9fbd03c7dd 2023-03-27 04:56:31,652 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/8845cc41983c4b1681d2c13c6afaed19 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/8845cc41983c4b1681d2c13c6afaed19 2023-03-27 04:56:31,654 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/fa374c80d0b0492c97861edbc118f435 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/A/fa374c80d0b0492c97861edbc118f435 2023-03-27 04:56:31,658 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/2c81c5e80c714eec98d747a831d91766 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/2c81c5e80c714eec98d747a831d91766 2023-03-27 04:56:31,660 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/7279d015135542c99b1d1defc87a3abf to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/7279d015135542c99b1d1defc87a3abf 2023-03-27 04:56:31,661 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/892adad167a34881bf4b241a23123673 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/892adad167a34881bf4b241a23123673 2023-03-27 04:56:31,662 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/a57b6e06947149b78d08b81eb56535bd to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/B/a57b6e06947149b78d08b81eb56535bd 2023-03-27 04:56:31,664 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/381ffcd3143c4828b93ce1459c562eff to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/381ffcd3143c4828b93ce1459c562eff 2023-03-27 04:56:31,666 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/6154e37b02954886bad4334892faa2a7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/6154e37b02954886bad4334892faa2a7 2023-03-27 04:56:31,667 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bd1803ca26c44b35bd4f3b97dca44f89 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/bd1803ca26c44b35bd4f3b97dca44f89 2023-03-27 04:56:31,669 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f0529fed43a24f8c88ce5b439115ce5c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/C/f0529fed43a24f8c88ce5b439115ce5c 2023-03-27 04:56:31,671 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/recovered.edits/674.seqid to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79/recovered.edits/674.seqid 2023-03-27 04:56:31,672 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/e7756c618a32a862ad613327bee74d79 2023-03-27 04:56:31,672 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-27 04:56:31,674 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:56:31,692 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-27 04:56:31,730 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-27 04:56:31,731 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:56:31,731 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-27 04:56:31,731 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679892991731"}]},"ts":"9223372036854775807"} 2023-03-27 04:56:31,734 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-27 04:56:31,734 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e7756c618a32a862ad613327bee74d79, NAME => 'TestAcidGuarantees,,1679892958957.e7756c618a32a862ad613327bee74d79.', STARTKEY => '', ENDKEY => ''}] 2023-03-27 04:56:31,734 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-27 04:56:31,734 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679892991734"}]},"ts":"9223372036854775807"} 2023-03-27 04:56:31,736 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-27 04:56:31,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-27 04:56:31,739 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:56:31,740 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 112 msec 2023-03-27 04:56:31,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-27 04:56:31,938 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-27 04:56:31,952 INFO [Listener at localhost.localdomain/43311] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=230 (was 207) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5749f0e9-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-686892593_15 at /127.0.0.1:36820 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5749f0e9-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5749f0e9-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5749f0e9-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase20:44923-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=524 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=533 (was 263) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=1026 (was 2024) 2023-03-27 04:56:31,963 INFO [Listener at localhost.localdomain/43311] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=230, OpenFileDescriptor=524, MaxFileDescriptor=60000, SystemLoadAverage=533, ProcessCount=167, AvailableMemoryMB=1026 2023-03-27 04:56:31,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-27 04:56:31,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-27 04:56:31,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-27 04:56:31,968 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-27 04:56:31,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-27 04:56:31,969 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-27 04:56:31,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-27 04:56:31,971 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:31,972 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293 empty. 2023-03-27 04:56:31,973 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:31,973 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-27 04:56:32,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-27 04:56:32,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-27 04:56:32,386 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-27 04:56:32,389 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 13f5a923b0a0364586ae63fe46bf7293, NAME => 'TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp 2023-03-27 04:56:32,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-27 04:56:32,799 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:56:32,800 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 13f5a923b0a0364586ae63fe46bf7293, disabling compactions & flushes 2023-03-27 04:56:32,800 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:32,800 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:32,800 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. after waiting 0 ms 2023-03-27 04:56:32,800 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:32,800 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:32,800 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:32,807 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-27 04:56:32,809 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679892992809"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679892992809"}]},"ts":"1679892992809"} 2023-03-27 04:56:32,811 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-27 04:56:32,812 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-27 04:56:32,812 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679892992812"}]},"ts":"1679892992812"} 2023-03-27 04:56:32,814 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-27 04:56:32,816 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=13f5a923b0a0364586ae63fe46bf7293, ASSIGN}] 2023-03-27 04:56:32,817 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=13f5a923b0a0364586ae63fe46bf7293, ASSIGN 2023-03-27 04:56:32,818 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=13f5a923b0a0364586ae63fe46bf7293, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,44923,1679892955505; forceNewPlan=false, retain=false 2023-03-27 04:56:32,969 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=13f5a923b0a0364586ae63fe46bf7293, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:32,969 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679892992969"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679892992969"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679892992969"}]},"ts":"1679892992969"} 2023-03-27 04:56:32,971 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:56:33,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-27 04:56:33,130 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:33,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 13f5a923b0a0364586ae63fe46bf7293, NAME => 'TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.', STARTKEY => '', ENDKEY => ''} 2023-03-27 04:56:33,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:33,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:56:33,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:33,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:33,134 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:33,137 DEBUG [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A 2023-03-27 04:56:33,137 DEBUG [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A 2023-03-27 04:56:33,137 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:56:33,137 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 13f5a923b0a0364586ae63fe46bf7293 columnFamilyName A 2023-03-27 04:56:33,138 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.HStore(310): Store=13f5a923b0a0364586ae63fe46bf7293/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:56:33,138 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:33,140 DEBUG [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B 2023-03-27 04:56:33,140 DEBUG [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B 2023-03-27 04:56:33,140 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:56:33,140 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 13f5a923b0a0364586ae63fe46bf7293 columnFamilyName B 2023-03-27 04:56:33,141 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.HStore(310): Store=13f5a923b0a0364586ae63fe46bf7293/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:56:33,141 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:33,142 DEBUG [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C 2023-03-27 04:56:33,142 DEBUG [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C 2023-03-27 04:56:33,143 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:56:33,143 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 13f5a923b0a0364586ae63fe46bf7293 columnFamilyName C 2023-03-27 04:56:33,143 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.HStore(310): Store=13f5a923b0a0364586ae63fe46bf7293/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:56:33,143 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:33,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:33,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:33,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-27 04:56:33,149 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:33,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-27 04:56:33,152 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 13f5a923b0a0364586ae63fe46bf7293; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70987393, jitterRate=0.057794585824012756}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-27 04:56:33,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:33,152 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., pid=18, masterSystemTime=1679892993123 2023-03-27 04:56:33,154 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:33,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:33,154 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=13f5a923b0a0364586ae63fe46bf7293, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:33,155 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679892993154"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679892993154"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679892993154"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679892993154"}]},"ts":"1679892993154"} 2023-03-27 04:56:33,157 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-27 04:56:33,158 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 in 185 msec 2023-03-27 04:56:33,159 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-27 04:56:33,159 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=13f5a923b0a0364586ae63fe46bf7293, ASSIGN in 342 msec 2023-03-27 04:56:33,160 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-27 04:56:33,160 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679892993160"}]},"ts":"1679892993160"} 2023-03-27 04:56:33,161 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-27 04:56:33,163 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-27 04:56:33,165 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1970 sec 2023-03-27 04:56:34,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-27 04:56:34,078 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-27 04:56:34,080 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x7306f12a to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:56:34,090 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31388307, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:56:34,093 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:56:34,099 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60392, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:56:34,101 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-27 04:56:34,104 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55474, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-27 04:56:34,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-27 04:56:34,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-27 04:56:34,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-27 04:56:34,538 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-27 04:56:34,541 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-27 04:56:34,541 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-27 04:56:34,548 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-27 04:56:34,561 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=13f5a923b0a0364586ae63fe46bf7293, REOPEN/MOVE}] 2023-03-27 04:56:34,562 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=13f5a923b0a0364586ae63fe46bf7293, REOPEN/MOVE 2023-03-27 04:56:34,563 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=13f5a923b0a0364586ae63fe46bf7293, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:34,563 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679892994563"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679892994563"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679892994563"}]},"ts":"1679892994563"} 2023-03-27 04:56:34,565 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:56:34,717 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:34,718 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 13f5a923b0a0364586ae63fe46bf7293, disabling compactions & flushes 2023-03-27 04:56:34,718 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:34,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:34,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. after waiting 0 ms 2023-03-27 04:56:34,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:34,725 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-27 04:56:34,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:34,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:34,727 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 13f5a923b0a0364586ae63fe46bf7293 to self. 2023-03-27 04:56:34,730 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:34,731 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=13f5a923b0a0364586ae63fe46bf7293, regionState=CLOSED 2023-03-27 04:56:34,731 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679892994731"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679892994731"}]},"ts":"1679892994731"} 2023-03-27 04:56:34,734 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-27 04:56:34,734 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 in 167 msec 2023-03-27 04:56:34,734 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=13f5a923b0a0364586ae63fe46bf7293, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,44923,1679892955505; forceNewPlan=false, retain=true 2023-03-27 04:56:34,885 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=13f5a923b0a0364586ae63fe46bf7293, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:34,885 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679892994885"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679892994885"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679892994885"}]},"ts":"1679892994885"} 2023-03-27 04:56:34,887 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:56:35,044 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:35,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 13f5a923b0a0364586ae63fe46bf7293, NAME => 'TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.', STARTKEY => '', ENDKEY => ''} 2023-03-27 04:56:35,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:35,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:56:35,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:35,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:35,049 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:35,050 DEBUG [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A 2023-03-27 04:56:35,050 DEBUG [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A 2023-03-27 04:56:35,051 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:56:35,054 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 13f5a923b0a0364586ae63fe46bf7293 columnFamilyName A 2023-03-27 04:56:35,056 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.HStore(310): Store=13f5a923b0a0364586ae63fe46bf7293/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:56:35,057 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:35,058 DEBUG [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B 2023-03-27 04:56:35,058 DEBUG [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B 2023-03-27 04:56:35,059 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:56:35,059 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 13f5a923b0a0364586ae63fe46bf7293 columnFamilyName B 2023-03-27 04:56:35,060 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.HStore(310): Store=13f5a923b0a0364586ae63fe46bf7293/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:56:35,060 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:35,062 DEBUG [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C 2023-03-27 04:56:35,062 DEBUG [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C 2023-03-27 04:56:35,062 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:56:35,064 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 13f5a923b0a0364586ae63fe46bf7293 columnFamilyName C 2023-03-27 04:56:35,065 INFO [StoreOpener-13f5a923b0a0364586ae63fe46bf7293-1] regionserver.HStore(310): Store=13f5a923b0a0364586ae63fe46bf7293/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:56:35,065 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:35,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:35,067 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:35,070 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-27 04:56:35,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:35,072 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 13f5a923b0a0364586ae63fe46bf7293; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65256737, jitterRate=-0.027598842978477478}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-27 04:56:35,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:35,073 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., pid=23, masterSystemTime=1679892995040 2023-03-27 04:56:35,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:35,075 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:35,075 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=13f5a923b0a0364586ae63fe46bf7293, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,075 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679892995075"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679892995075"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679892995075"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679892995075"}]},"ts":"1679892995075"} 2023-03-27 04:56:35,079 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-27 04:56:35,079 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 in 190 msec 2023-03-27 04:56:35,080 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-27 04:56:35,081 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=13f5a923b0a0364586ae63fe46bf7293, REOPEN/MOVE in 518 msec 2023-03-27 04:56:35,084 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-27 04:56:35,084 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 535 msec 2023-03-27 04:56:35,086 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 969 msec 2023-03-27 04:56:35,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-27 04:56:35,096 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x5e3bbbd0 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:56:35,101 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23640038, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:56:35,102 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x2d781def to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:56:35,108 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@df763e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:56:35,108 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x3b3311fc to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:56:35,115 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5649f410, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:56:35,116 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x7acc1387 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:56:35,129 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@633c81ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:56:35,129 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x7be739cc to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:56:35,141 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61ad5d1c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:56:35,142 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x486fba63 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:56:35,149 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43a14bcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:56:35,149 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x50feca84 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:56:35,153 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6577ce5a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:56:35,154 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x0bd79dff to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:56:35,158 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d5bdceb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:56:35,159 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x52902392 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:56:35,163 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@651f745a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:56:35,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:35,181 DEBUG [hconnection-0x3efea84e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:56:35,181 DEBUG [hconnection-0x2d7d2aa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:56:35,181 DEBUG [hconnection-0x310c5c0f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:56:35,182 DEBUG [hconnection-0xa2f2c2a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:56:35,182 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60406, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:56:35,183 DEBUG [hconnection-0x421b140f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:56:35,185 DEBUG [hconnection-0x8d23ab7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:56:35,187 DEBUG [hconnection-0x6b614678-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:56:35,187 DEBUG [hconnection-0x17b34800-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:56:35,187 DEBUG [hconnection-0xe9ee97c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:56:35,191 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60418, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:56:35,191 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60414, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:56:35,192 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60422, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:56:35,192 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60430, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:56:35,194 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60440, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:56:35,195 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60460, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:56:35,194 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60424, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:56:35,195 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60456, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:56:35,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:35,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:35,202 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:35,203 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:35,203 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:35,203 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:35,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:35,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:35,205 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,205 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,205 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:35,207 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:35,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:35,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:35,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:35,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:35,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:35,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:35,208 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:35,208 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:35,208 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:35,208 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:35,209 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:35,209 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:35,209 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:35,212 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-27 04:56:35,214 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:35,214 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:35,214 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:35,214 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:35,214 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:35,214 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:35,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:35,217 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:35,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679892995226, max timestamp=1679892995234Num uniques 859; 2023-03-27 04:56:35,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:35,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679892995226, max timestamp=1679892995234Num uniques 859; 2023-03-27 04:56:35,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:35,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679892995226, max timestamp=1679892995234Num uniques 859; 2023-03-27 04:56:35,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:35,268 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327e8b711281f1849a1b980964fca43f345 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e8b711281f1849a1b980964fca43f345 2023-03-27 04:56:35,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679892995261, max timestamp=1679892995265Num uniques 1018; 2023-03-27 04:56:35,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679892995261, max timestamp=1679892995265Num uniques 1018; 2023-03-27 04:56:35,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-27 04:56:35,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-27 04:56:35,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679892995261, max timestamp=1679892995265Num uniques 1018; 2023-03-27 04:56:35,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-03-27 04:56:35,288 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=12, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/0a6dec5e88144493baec26c739d2597c 2023-03-27 04:56:35,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679892995287, max timestamp=1679892995299Num uniques 1027; 2023-03-27 04:56:35,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-27 04:56:35,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679892995287, max timestamp=1679892995299Num uniques 1027; 2023-03-27 04:56:35,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-27 04:56:35,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679892995287, max timestamp=1679892995300Num uniques 1009; 2023-03-27 04:56:35,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=12682, suffix heap size=132904,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=277 2023-03-27 04:56:35,318 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/6bbd2cb2203b4fd39c39cfd31ff40ab7 2023-03-27 04:56:35,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893055350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893055350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60418 deadline: 1679893055351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60422 deadline: 1679893055352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60440 deadline: 1679893055352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893055461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893055461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60422 deadline: 1679893055461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60418 deadline: 1679893055466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60440 deadline: 1679893055466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893055664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893055664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60422 deadline: 1679893055664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60440 deadline: 1679893055669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60418 deadline: 1679893055669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,731 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/147f964d333b4c2186fc691938f6417b 2023-03-27 04:56:35,737 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/0a6dec5e88144493baec26c739d2597c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0a6dec5e88144493baec26c739d2597c 2023-03-27 04:56:35,741 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0a6dec5e88144493baec26c739d2597c, entries=150, sequenceid=12, filesize=25.1 K 2023-03-27 04:56:35,742 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/6bbd2cb2203b4fd39c39cfd31ff40ab7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/6bbd2cb2203b4fd39c39cfd31ff40ab7 2023-03-27 04:56:35,748 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/6bbd2cb2203b4fd39c39cfd31ff40ab7, entries=150, sequenceid=12, filesize=11.6 K 2023-03-27 04:56:35,751 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/147f964d333b4c2186fc691938f6417b as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/147f964d333b4c2186fc691938f6417b 2023-03-27 04:56:35,763 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/147f964d333b4c2186fc691938f6417b, entries=150, sequenceid=12, filesize=11.6 K 2023-03-27 04:56:35,764 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 13f5a923b0a0364586ae63fe46bf7293 in 553ms, sequenceid=12, compaction requested=false 2023-03-27 04:56:35,764 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:35,764 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:35,764 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:35,764 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:35,764 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,764 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:35,764 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:35,766 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:35,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:35,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:35,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:35,767 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:35,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:35,768 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:35,768 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:35,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:35,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:35,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76f185dd[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:35,768 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:35,768 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:35,769 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:35,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:35,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:35,770 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:35,770 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:35,770 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:35,771 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,771 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:35,771 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:35,772 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:35,772 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:35,772 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:35,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:35,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:35,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:35,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:35,775 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:35,775 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:35,775 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:35,775 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:35,775 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:35,778 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:35,778 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:35,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:35,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:35,778 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:35,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:35,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:35,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:35,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:35,779 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:35,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:35,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:35,780 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:35,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:35,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:35,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:35,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:35,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:35,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:35,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:35,783 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:35,783 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:35,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:35,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:35,782 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:35,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:35,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:35,783 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:35,783 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:35,883 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:35,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:35,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:35,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:35,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:35,888 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:35,888 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:35,888 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:35,888 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:35,888 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:35,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:35,889 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:35,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:35,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:35,890 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,890 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:35,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:35,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:35,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:35,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:35,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:35,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:35,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:35,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:35,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:35,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:35,902 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:35,902 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:35,902 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:35,902 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:35,902 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=147.60 KB heapSize=413.77 KB 2023-03-27 04:56:35,903 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:35,903 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:35,903 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:35,903 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:35,903 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:35,903 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:35,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:35,970 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:35,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60422 deadline: 1679893055971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893055975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60440 deadline: 1679893055975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893055975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:35,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:35,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60418 deadline: 1679893055975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:36,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:36,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60422 deadline: 1679893056076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:36,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:36,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893056078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:36,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:36,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893056078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:36,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:36,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60422 deadline: 1679893056281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:36,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:36,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:36,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893056285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:36,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893056285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:36,334 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032733e4d037e6c44c0dbe37f37c837a71d2 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032733e4d037e6c44c0dbe37f37c837a71d2 2023-03-27 04:56:36,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:36,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60418 deadline: 1679893056478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:36,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:36,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60440 deadline: 1679893056481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:36,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:36,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60422 deadline: 1679893056586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:36,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:36,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893056590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:36,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:36,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893056590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:36,707 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-27 04:56:36,741 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/c7ba3c92e24940aba1c6c7293dec42f8 2023-03-27 04:56:37,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:37,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60422 deadline: 1679893057087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:37,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893057097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:37,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893057099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,174 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/ad978f4bec9e456ba092c2655bc0d5b3 2023-03-27 04:56:37,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:37,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60418 deadline: 1679893057483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:37,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60440 deadline: 1679893057485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,589 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/762988dfac854405b7d7ed918ac7abee 2023-03-27 04:56:37,594 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/c7ba3c92e24940aba1c6c7293dec42f8 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/c7ba3c92e24940aba1c6c7293dec42f8 2023-03-27 04:56:37,599 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/c7ba3c92e24940aba1c6c7293dec42f8, entries=150, sequenceid=83, filesize=25.1 K 2023-03-27 04:56:37,601 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/ad978f4bec9e456ba092c2655bc0d5b3 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/ad978f4bec9e456ba092c2655bc0d5b3 2023-03-27 04:56:37,610 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/ad978f4bec9e456ba092c2655bc0d5b3, entries=150, sequenceid=83, filesize=11.6 K 2023-03-27 04:56:37,611 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/762988dfac854405b7d7ed918ac7abee as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/762988dfac854405b7d7ed918ac7abee 2023-03-27 04:56:37,616 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/762988dfac854405b7d7ed918ac7abee, entries=150, sequenceid=83, filesize=11.6 K 2023-03-27 04:56:37,617 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~413.72 KB/423648, currentSize=33.54 KB/34350 for 13f5a923b0a0364586ae63fe46bf7293 in 1715ms, sequenceid=83, compaction requested=false 2023-03-27 04:56:37,617 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:37,617 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:37,617 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:37,617 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:37,617 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,618 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:37,618 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:37,619 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,619 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:37,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:37,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:37,619 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:37,619 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:37,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:37,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:37,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:37,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:37,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:37,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5973a26a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:37,620 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:37,620 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:37,621 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:37,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:37,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:37,621 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:37,621 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:37,621 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:37,621 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:37,622 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:37,622 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:37,622 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:37,623 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:37,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:37,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:37,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:37,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:37,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:37,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:37,626 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:37,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:37,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:37,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:37,626 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:37,627 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:37,627 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:37,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:37,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:37,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:37,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:37,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:37,628 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:37,629 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:37,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:37,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:37,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:37,632 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:37,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:37,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,648 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:37,648 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:37,648 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:37,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:37,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:37,648 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:37,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:37,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:37,648 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:37,648 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:37,648 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,648 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:37,648 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:37,648 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:37,648 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:37,648 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:37,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:37,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:37,749 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:37,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:37,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:37,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:37,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:37,752 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:37,752 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:37,752 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:37,752 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:37,752 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:37,752 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:37,753 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:37,753 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:37,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:37,754 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:37,754 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:37,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:37,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:37,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:37,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:37,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:37,770 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:37,770 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:37,770 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:37,770 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:37,770 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:37,770 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:37,770 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:37,770 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:37,770 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:37,771 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-27 04:56:37,771 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:37,771 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:37,771 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:37,771 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:37,771 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:37,771 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:38,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:38,107 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:38,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1679892998109, max timestamp=1679892998124Num uniques 859; 2023-03-27 04:56:38,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-27 04:56:38,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1679892998109, max timestamp=1679892998124Num uniques 859; 2023-03-27 04:56:38,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679892998109, max timestamp=1679892998124Num uniques 859; 2023-03-27 04:56:38,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-27 04:56:38,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-27 04:56:38,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:38,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893058147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:38,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:38,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893058148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:38,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:38,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60422 deadline: 1679893058150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:38,188 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032740eab22e00d8473bbb5339e05f5a6b01 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032740eab22e00d8473bbb5339e05f5a6b01 2023-03-27 04:56:38,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:38,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893058251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:38,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:38,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893058252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:38,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:38,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60422 deadline: 1679893058255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:38,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:38,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893058454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:38,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:38,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893058456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:38,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:38,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60422 deadline: 1679893058460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:38,594 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=91, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/675db2aac9254823825689568e21cb4b 2023-03-27 04:56:38,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:38,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893058757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:38,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:38,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893058758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:38,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:38,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60422 deadline: 1679893058765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,007 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/b7e5b545c2fa4d719c64312b76fd05ce 2023-03-27 04:56:39,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:39,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893059261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:39,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893059265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:39,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60422 deadline: 1679893059270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,418 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/e1f60d59b4e2409686e6c8ecb8be45ff 2023-03-27 04:56:39,423 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/675db2aac9254823825689568e21cb4b as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/675db2aac9254823825689568e21cb4b 2023-03-27 04:56:39,427 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/675db2aac9254823825689568e21cb4b, entries=150, sequenceid=91, filesize=25.1 K 2023-03-27 04:56:39,428 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/b7e5b545c2fa4d719c64312b76fd05ce as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/b7e5b545c2fa4d719c64312b76fd05ce 2023-03-27 04:56:39,432 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/b7e5b545c2fa4d719c64312b76fd05ce, entries=150, sequenceid=91, filesize=11.6 K 2023-03-27 04:56:39,433 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/e1f60d59b4e2409686e6c8ecb8be45ff as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e1f60d59b4e2409686e6c8ecb8be45ff 2023-03-27 04:56:39,442 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e1f60d59b4e2409686e6c8ecb8be45ff, entries=150, sequenceid=91, filesize=11.6 K 2023-03-27 04:56:39,443 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=156.54 KB/160300 for 13f5a923b0a0364586ae63fe46bf7293 in 1672ms, sequenceid=91, compaction requested=true 2023-03-27 04:56:39,443 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:39,443 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:39,443 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:39,443 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:39,443 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,443 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:39,443 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:39,444 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,444 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:39,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:39,445 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,445 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:39,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:39,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:39,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:39,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:39,446 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:39,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b50b372[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:39,446 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,447 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,447 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:39,447 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:39,447 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:39,447 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,447 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:39,449 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,449 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:39,449 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:39,449 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:39,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:39,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:39,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:39,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:39,453 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:39,453 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:39,453 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:39,453 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:39,453 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:39,454 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,454 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:39,454 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:39,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:39,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,456 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:39,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,457 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:39,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,458 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:39,458 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:39,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:39,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:39,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:39,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:39,459 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:39,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:39,459 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:39,459 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:39,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:39,504 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=163.25 KB heapSize=451 KB 2023-03-27 04:56:39,504 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:39,504 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:39,504 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:39,504 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:39,504 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:39,504 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:39,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:39,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60418 deadline: 1679893059510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:39,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60440 deadline: 1679893059511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,559 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:39,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:39,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:39,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:39,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:39,563 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:39,563 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:39,563 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:39,563 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:39,563 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,563 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,564 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:39,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:39,564 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,564 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,565 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:39,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:39,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:39,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:39,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:39,566 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:39,566 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:39,566 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:39,570 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:39,570 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:39,570 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:39,570 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:39,570 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:39,570 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:39,570 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:39,570 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:39,570 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,570 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:39,570 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:39,577 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,577 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:39,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:39,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:39,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:39,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:39,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:39,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:39,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@557c0b45[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:39,579 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:39,579 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,581 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,581 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:39,581 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:39,581 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:39,581 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,581 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:39,582 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:39,582 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:39,582 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:39,582 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:39,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:39,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:39,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:39,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:39,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:39,587 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:39,587 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:39,587 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:39,587 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:39,587 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:39,594 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,594 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:39,594 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:39,595 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:39,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:39,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,597 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:39,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,602 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,602 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,602 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,602 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,602 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,602 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:39,603 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:39,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:39,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:39,603 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:39,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:39,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:39,603 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:39,603 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:39,603 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:39,603 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:39,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:39,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:39,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60418 deadline: 1679893059615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:39,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60440 deadline: 1679893059617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,703 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:39,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:39,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:39,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:39,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:39,706 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:39,706 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:39,706 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:39,706 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:39,707 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,707 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,708 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,708 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:39,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:39,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:39,708 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,708 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:39,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:39,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:39,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:39,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:39,710 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:39,710 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:39,710 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:39,710 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:39,710 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:39,710 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:39,710 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:39,710 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:39,710 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:39,711 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:39,711 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:39,711 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,711 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:39,711 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:39,712 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,712 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:39,712 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,712 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:39,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:39,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f59a35a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:39,713 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:39,714 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,717 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,717 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:39,717 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:39,717 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:39,718 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,718 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:39,718 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:39,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:39,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:39,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:39,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:39,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:39,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:39,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:39,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:39,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:39,721 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:39,721 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:39,721 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:39,721 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:39,721 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:39,722 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,722 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:39,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:39,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,722 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:39,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:39,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:39,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:39,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:39,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:39,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,725 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:39,725 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:39,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:39,725 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:39,725 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,725 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:39,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:39,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:39,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:39,725 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:39,725 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,726 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:39,726 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,726 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:39,726 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,726 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:39,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60418 deadline: 1679893059818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:39,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60440 deadline: 1679893059824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,826 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:39,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:39,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:39,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:39,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:39,829 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:39,829 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:39,829 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:39,829 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:39,829 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,829 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,830 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,830 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:39,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:39,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:39,830 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,830 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:39,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:39,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:39,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:39,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:39,831 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:39,831 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:39,831 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:39,832 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:39,832 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:39,832 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:39,832 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:39,832 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:39,832 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:39,832 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:39,832 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:39,832 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,832 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:39,832 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:39,833 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,833 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:39,833 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,833 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:39,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:39,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b7d3429[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:39,834 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:39,834 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,835 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,835 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:39,835 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:39,835 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:39,835 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,835 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:39,836 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:39,836 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:39,836 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,836 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:39,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:39,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:39,838 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:39,838 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:39,838 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:39,838 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:39,838 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:39,838 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:39,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:39,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:39,839 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:39,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:39,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,842 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,845 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:39,845 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:39,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:39,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:39,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:39,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:39,845 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:39,845 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,845 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,845 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,845 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:39,845 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,845 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:39,845 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,845 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:39,845 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:39,921 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303278e3b2f1b373b4d54b2c5a32c7c14552d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303278e3b2f1b373b4d54b2c5a32c7c14552d 2023-03-27 04:56:39,945 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:39,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:39,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:39,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:39,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:39,949 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:39,949 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:39,949 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:39,949 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:39,949 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,949 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,950 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:39,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:39,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:39,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:39,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:39,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:39,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:39,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:39,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:39,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:39,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:39,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:39,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:39,951 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:39,951 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:39,951 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:39,951 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:39,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:39,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:39,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:39,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:39,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,952 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:39,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:39,953 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,953 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:39,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:39,954 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:39,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4518d117[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:39,954 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,954 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,954 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:39,954 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:39,954 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:39,955 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,955 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:39,955 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,955 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:39,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:39,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:39,955 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:39,955 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:39,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:39,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:39,957 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:39,957 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:39,957 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:39,957 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:39,957 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:39,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:39,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:39,959 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:39,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,960 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:39,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:39,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:39,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:39,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:39,962 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:39,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:39,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:39,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:39,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:39,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,962 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:39,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:39,962 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:39,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:39,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:40,062 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:40,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:40,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:40,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:40,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:40,065 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:40,065 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:40,065 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:40,065 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:40,066 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,066 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,066 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,066 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:40,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:40,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:40,067 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,067 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:40,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:40,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:40,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:40,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:40,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:40,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:40,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:40,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:40,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:40,068 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:40,068 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:40,068 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:40,068 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:40,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:40,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:40,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:40,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:40,069 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,069 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:40,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:40,069 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,069 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:40,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:40,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:40,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:40,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:40,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@700bccd2[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:40,071 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:40,071 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,071 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,072 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:40,072 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:40,072 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:40,073 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,073 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:40,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:40,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:40,073 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:40,073 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:40,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:40,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:40,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:40,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:40,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:40,078 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,078 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:40,078 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:40,078 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:40,078 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:40,078 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:40,078 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:40,079 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,079 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:40,079 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:40,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:40,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:40,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:40,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:40,081 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:40,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:40,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,084 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,094 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:40,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:40,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:40,094 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,095 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,095 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,095 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,095 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:40,095 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:40,094 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:40,095 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:40,095 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,095 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:40,095 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:40,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:40,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:40,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:40,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60418 deadline: 1679893060122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:40,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60440 deadline: 1679893060125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,195 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:40,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:40,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:40,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:40,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:40,198 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:40,198 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:40,198 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:40,198 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:40,198 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,198 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,199 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:40,199 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:40,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:40,199 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,199 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:40,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:40,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:40,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:40,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:40,200 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:40,200 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:40,200 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:40,200 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:40,200 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:40,200 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:40,200 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:40,200 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:40,200 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:40,200 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:40,200 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:40,200 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,200 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:40,200 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:40,201 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,201 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:40,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:40,201 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,201 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:40,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:40,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:40,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:40,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:40,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12aae2fa[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:40,203 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:40,203 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,204 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,204 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:40,204 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:40,204 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:40,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:40,204 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,204 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:40,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:40,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:40,204 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:40,204 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:40,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:40,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:40,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:40,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:40,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:40,206 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:40,206 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:40,206 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:40,206 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:40,206 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:40,207 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,207 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:40,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:40,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:40,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,208 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:40,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:40,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:40,208 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:40,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,209 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:40,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,211 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,211 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,211 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,211 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:40,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:40,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:40,211 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:40,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:40,211 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:40,211 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,212 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:40,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:40,212 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:40,212 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,212 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,212 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:40,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:40,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:40,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893060271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:40,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893060273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:40,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60422 deadline: 1679893060279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,312 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:40,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:40,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:40,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:40,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:40,315 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:40,315 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:40,315 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:40,315 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:40,316 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,316 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,316 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:40,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:40,317 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:40,317 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,317 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:40,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:40,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:40,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:40,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:40,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:40,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:40,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:40,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:40,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:40,318 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:40,318 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:40,318 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:40,318 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:40,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:40,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:40,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:40,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:40,319 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,319 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:40,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:40,320 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,320 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:40,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:40,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:40,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:40,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:40,321 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:40,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10f440fa[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:40,321 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,322 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,322 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:40,322 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,322 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:40,322 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:40,322 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:40,323 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,323 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:40,323 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:40,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:40,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:40,323 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:40,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:40,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:40,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:40,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:40,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:40,325 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:40,325 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:40,325 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:40,325 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:40,325 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:40,326 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,326 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:40,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:40,326 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:40,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:40,326 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:40,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:40,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,327 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:40,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:40,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,329 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,329 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,329 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,329 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,329 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,329 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:40,329 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,329 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:40,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:40,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:40,329 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:40,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:40,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:40,329 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:40,329 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:40,329 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:40,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:40,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:40,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:40,335 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=128, memsize=55.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/349b2bac84194ef78f3a25267b18d53b 2023-03-27 04:56:40,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/941011a49891441e9db5de488c755db7 2023-03-27 04:56:40,360 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/56c33b2caf8c403ebbf240782a62f8b7 2023-03-27 04:56:40,364 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/349b2bac84194ef78f3a25267b18d53b as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/349b2bac84194ef78f3a25267b18d53b 2023-03-27 04:56:40,368 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/349b2bac84194ef78f3a25267b18d53b, entries=200, sequenceid=128, filesize=31.9 K 2023-03-27 04:56:40,369 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/941011a49891441e9db5de488c755db7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/941011a49891441e9db5de488c755db7 2023-03-27 04:56:40,373 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/941011a49891441e9db5de488c755db7, entries=150, sequenceid=128, filesize=11.6 K 2023-03-27 04:56:40,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/56c33b2caf8c403ebbf240782a62f8b7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/56c33b2caf8c403ebbf240782a62f8b7 2023-03-27 04:56:40,377 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/56c33b2caf8c403ebbf240782a62f8b7, entries=150, sequenceid=128, filesize=11.6 K 2023-03-27 04:56:40,378 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~163.25 KB/167170, heapSize ~450.95 KB/461776, currentSize=26.84 KB/27480 for 13f5a923b0a0364586ae63fe46bf7293 in 874ms, sequenceid=128, compaction requested=true 2023-03-27 04:56:40,378 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:40,378 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:40,378 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:56:40,379 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:40,378 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:56:40,379 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:40,380 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 109924 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:56:40,380 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:56:40,380 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 13f5a923b0a0364586ae63fe46bf7293/A is initiating minor compaction (all files) 2023-03-27 04:56:40,380 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 13f5a923b0a0364586ae63fe46bf7293/B is initiating minor compaction (all files) 2023-03-27 04:56:40,380 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 13f5a923b0a0364586ae63fe46bf7293/A in TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:40,380 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 13f5a923b0a0364586ae63fe46bf7293/B in TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:40,380 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/6bbd2cb2203b4fd39c39cfd31ff40ab7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/ad978f4bec9e456ba092c2655bc0d5b3, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/b7e5b545c2fa4d719c64312b76fd05ce, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/941011a49891441e9db5de488c755db7] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp, totalSize=46.4 K 2023-03-27 04:56:40,380 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0a6dec5e88144493baec26c739d2597c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/c7ba3c92e24940aba1c6c7293dec42f8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/675db2aac9254823825689568e21cb4b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/349b2bac84194ef78f3a25267b18d53b] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp, totalSize=107.3 K 2023-03-27 04:56:40,380 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 6bbd2cb2203b4fd39c39cfd31ff40ab7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679892995204 2023-03-27 04:56:40,380 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 0a6dec5e88144493baec26c739d2597c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679892995204 2023-03-27 04:56:40,381 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting ad978f4bec9e456ba092c2655bc0d5b3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679892995335 2023-03-27 04:56:40,381 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting c7ba3c92e24940aba1c6c7293dec42f8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679892995335 2023-03-27 04:56:40,381 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting b7e5b545c2fa4d719c64312b76fd05ce, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1679892995969 2023-03-27 04:56:40,381 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 675db2aac9254823825689568e21cb4b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1679892995969 2023-03-27 04:56:40,381 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 941011a49891441e9db5de488c755db7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1679892998140 2023-03-27 04:56:40,381 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 349b2bac84194ef78f3a25267b18d53b, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1679892998140 2023-03-27 04:56:40,392 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 13f5a923b0a0364586ae63fe46bf7293#B#compaction#138 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:40,395 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 13f5a923b0a0364586ae63fe46bf7293#A#compaction#137 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:40,429 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:40,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:40,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:40,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:40,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:40,436 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:40,436 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:40,436 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:40,436 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:40,437 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,437 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,438 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:40,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:40,438 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:40,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:40,439 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,439 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:40,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:40,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:40,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:40,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:40,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:40,442 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:40,442 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:40,442 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:40,442 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:40,442 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:40,442 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:40,443 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:40,443 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-27 04:56:40,444 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/87c0a7193236450db9884f33e6f85d9b as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/87c0a7193236450db9884f33e6f85d9b 2023-03-27 04:56:40,444 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:40,444 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:40,444 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:40,444 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:40,444 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:40,445 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:40,454 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 13f5a923b0a0364586ae63fe46bf7293/B of 13f5a923b0a0364586ae63fe46bf7293 into 87c0a7193236450db9884f33e6f85d9b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:40,454 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:40,454 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., storeName=13f5a923b0a0364586ae63fe46bf7293/B, priority=12, startTime=1679893000378; duration=0sec 2023-03-27 04:56:40,454 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:40,454 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:56:40,456 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:56:40,456 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 13f5a923b0a0364586ae63fe46bf7293/C is initiating minor compaction (all files) 2023-03-27 04:56:40,456 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 13f5a923b0a0364586ae63fe46bf7293/C in TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:40,456 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/147f964d333b4c2186fc691938f6417b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/762988dfac854405b7d7ed918ac7abee, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e1f60d59b4e2409686e6c8ecb8be45ff, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/56c33b2caf8c403ebbf240782a62f8b7] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp, totalSize=46.4 K 2023-03-27 04:56:40,457 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 147f964d333b4c2186fc691938f6417b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679892995204 2023-03-27 04:56:40,457 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 762988dfac854405b7d7ed918ac7abee, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679892995335 2023-03-27 04:56:40,457 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting e1f60d59b4e2409686e6c8ecb8be45ff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1679892995969 2023-03-27 04:56:40,458 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 56c33b2caf8c403ebbf240782a62f8b7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1679892998140 2023-03-27 04:56:40,467 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 13f5a923b0a0364586ae63fe46bf7293#C#compaction#140 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:40,467 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327c41f48b84e3e47efafe4975f525496de to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327c41f48b84e3e47efafe4975f525496de 2023-03-27 04:56:40,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:40,630 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:40,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679893000640, max timestamp=1679893000675Num uniques 859; 2023-03-27 04:56:40,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679893000640, max timestamp=1679893000675Num uniques 859; 2023-03-27 04:56:40,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679893000640, max timestamp=1679893000675Num uniques 859; 2023-03-27 04:56:40,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-27 04:56:40,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-27 04:56:40,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-27 04:56:40,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679893000690, max timestamp=1679893000704Num uniques 1118; 2023-03-27 04:56:40,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-27 04:56:40,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679893000690, max timestamp=1679893000704Num uniques 1118; 2023-03-27 04:56:40,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-27 04:56:40,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679893000690, max timestamp=1679893000704Num uniques 1118; 2023-03-27 04:56:40,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-27 04:56:40,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:40,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60440 deadline: 1679893060753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679893000728, max timestamp=1679893000755Num uniques 1127; 2023-03-27 04:56:40,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-03-27 04:56:40,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679893000728, max timestamp=1679893000755Num uniques 1127; 2023-03-27 04:56:40,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-03-27 04:56:40,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679893000742, max timestamp=1679893000755Num uniques 1127; 2023-03-27 04:56:40,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-27 04:56:40,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679893000784, max timestamp=1679893000830Num uniques 1186; 2023-03-27 04:56:40,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=8516, suffix heap size=151312,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=186 2023-03-27 04:56:40,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679893000784, max timestamp=1679893000830Num uniques 1036; 2023-03-27 04:56:40,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-03-27 04:56:40,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679893000784, max timestamp=1679893000830Num uniques 1186; 2023-03-27 04:56:40,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=8516, suffix heap size=151312,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=186 2023-03-27 04:56:40,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679893000861, max timestamp=1679893000868Num uniques 1045; 2023-03-27 04:56:40,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-27 04:56:40,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679893000861, max timestamp=1679893000868Num uniques 1009; 2023-03-27 04:56:40,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679893000861, max timestamp=1679893000868Num uniques 1045; 2023-03-27 04:56:40,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11220, suffix heap size=132904,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=245 2023-03-27 04:56:40,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-27 04:56:40,884 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=135, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/0d4b69010b164bf28dd841228b8c40c1 2023-03-27 04:56:40,884 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/cb7f36d0a7a04284b6dc25e9ed62f221 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/cb7f36d0a7a04284b6dc25e9ed62f221 2023-03-27 04:56:40,890 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 13f5a923b0a0364586ae63fe46bf7293/C of 13f5a923b0a0364586ae63fe46bf7293 into cb7f36d0a7a04284b6dc25e9ed62f221(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:40,890 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:40,890 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., storeName=13f5a923b0a0364586ae63fe46bf7293/C, priority=12, startTime=1679893000379; duration=0sec 2023-03-27 04:56:40,891 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:40,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679893000881, max timestamp=1679893000896Num uniques 1104; 2023-03-27 04:56:40,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-27 04:56:40,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679893000885, max timestamp=1679893000897Num uniques 1104; 2023-03-27 04:56:40,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-27 04:56:40,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679893000885, max timestamp=1679893000896Num uniques 1104; 2023-03-27 04:56:40,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=6870, suffix heap size=142784,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=150 2023-03-27 04:56:40,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:40,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60418 deadline: 1679893060956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679893000947, max timestamp=1679893000954Num uniques 1013; 2023-03-27 04:56:40,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679893000947, max timestamp=1679893000954Num uniques 1009; 2023-03-27 04:56:40,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679893000947, max timestamp=1679893000954Num uniques 1113; 2023-03-27 04:56:40,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46208, new segment data size=9748, suffix heap size=132904,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=213 2023-03-27 04:56:40,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-27 04:56:40,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-27 04:56:40,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679893000980, max timestamp=1679893000993Num uniques 1072; 2023-03-27 04:56:40,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:40,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-27 04:56:40,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60440 deadline: 1679893060995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:40,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679893000980, max timestamp=1679893000993Num uniques 1072; 2023-03-27 04:56:40,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-27 04:56:40,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679893000980, max timestamp=1679893000993Num uniques 1072; 2023-03-27 04:56:40,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-27 04:56:41,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679893001076, max timestamp=1679893001090Num uniques 1081; 2023-03-27 04:56:41,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-27 04:56:41,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679893001076, max timestamp=1679893001090Num uniques 1081; 2023-03-27 04:56:41,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-27 04:56:41,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679893001076, max timestamp=1679893001090Num uniques 1081; 2023-03-27 04:56:41,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-27 04:56:41,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679893001112, max timestamp=1679893001120Num uniques 1090; 2023-03-27 04:56:41,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-03-27 04:56:41,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679893001112, max timestamp=1679893001120Num uniques 1090; 2023-03-27 04:56:41,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-03-27 04:56:41,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679893001112, max timestamp=1679893001120Num uniques 1090; 2023-03-27 04:56:41,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-03-27 04:56:41,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15984, totalHeapSize=36536, min timestamp=1679893001128, max timestamp=1679893001149Num uniques 1099; 2023-03-27 04:56:41,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=15984, suffix heap size=142264,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=349 2023-03-27 04:56:41,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15984, totalHeapSize=36536, min timestamp=1679893001128, max timestamp=1679893001149Num uniques 1099; 2023-03-27 04:56:41,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15984, totalHeapSize=36536, min timestamp=1679893001128, max timestamp=1679893001149Num uniques 1099; 2023-03-27 04:56:41,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=15984, suffix heap size=142264,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=349 2023-03-27 04:56:41,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=15984, suffix heap size=142264,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=349 2023-03-27 04:56:41,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:41,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893061196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:41,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893061197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,251 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/e4c2f0686a3a48639f989c12cf79e855 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/e4c2f0686a3a48639f989c12cf79e855 2023-03-27 04:56:41,256 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 13f5a923b0a0364586ae63fe46bf7293/A of 13f5a923b0a0364586ae63fe46bf7293 into e4c2f0686a3a48639f989c12cf79e855(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:41,256 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:41,256 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., storeName=13f5a923b0a0364586ae63fe46bf7293/A, priority=12, startTime=1679893000378; duration=0sec 2023-03-27 04:56:41,256 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:41,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:41,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893061298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:41,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893061299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,317 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/29cbccac02854bb0985a808e65c5fbc0 2023-03-27 04:56:41,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:41,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:41,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893061501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893061501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,727 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/a71d644a1c0b4bef976507b42b581b83 2023-03-27 04:56:41,732 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/0d4b69010b164bf28dd841228b8c40c1 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0d4b69010b164bf28dd841228b8c40c1 2023-03-27 04:56:41,738 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0d4b69010b164bf28dd841228b8c40c1, entries=150, sequenceid=135, filesize=25.3 K 2023-03-27 04:56:41,739 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/29cbccac02854bb0985a808e65c5fbc0 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/29cbccac02854bb0985a808e65c5fbc0 2023-03-27 04:56:41,746 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/29cbccac02854bb0985a808e65c5fbc0, entries=150, sequenceid=135, filesize=11.8 K 2023-03-27 04:56:41,747 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/a71d644a1c0b4bef976507b42b581b83 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/a71d644a1c0b4bef976507b42b581b83 2023-03-27 04:56:41,755 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/a71d644a1c0b4bef976507b42b581b83, entries=150, sequenceid=135, filesize=11.8 K 2023-03-27 04:56:41,757 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 13f5a923b0a0364586ae63fe46bf7293 in 1314ms, sequenceid=135, compaction requested=false 2023-03-27 04:56:41,757 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:41,757 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:41,757 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:41,757 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:41,757 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,758 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:41,758 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:41,759 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,759 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:41,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:41,759 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,759 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:41,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:41,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:41,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:41,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:41,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:41,761 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:41,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1367ab9a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:41,761 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,761 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,761 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:41,761 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:41,761 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:41,761 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,761 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:41,762 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,762 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:41,762 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:41,762 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:41,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:41,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:41,764 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:41,764 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:41,764 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:41,764 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:41,764 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:41,765 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,765 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:41,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:41,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,765 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:41,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:41,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:41,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:41,766 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:41,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:41,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:41,767 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:41,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:41,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,768 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:41,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:41,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:41,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:41,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:41,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:41,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:41,769 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:41,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:41,769 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:41,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:41,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:41,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:41,805 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=167.72 KB heapSize=467.48 KB 2023-03-27 04:56:41,806 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:41,806 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-27 04:56:41,806 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:41,806 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-27 04:56:41,806 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:41,806 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-27 04:56:41,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:41,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893061810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:41,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893061812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,869 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:41,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:41,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:41,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:41,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:41,871 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:41,871 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:41,871 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:41,871 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:41,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:41,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,872 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:41,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:41,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:41,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,873 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:41,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:41,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:41,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:41,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:41,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:41,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:41,873 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:41,873 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:41,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:41,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:41,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:41,874 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:41,874 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:41,874 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:41,874 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:41,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:41,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:41,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:41,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:41,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,875 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:41,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:41,875 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,875 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:41,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:41,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:41,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:41,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:41,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:41,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3da48878[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:41,876 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:41,876 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,877 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,877 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:41,877 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:41,877 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:41,877 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,877 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:41,878 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,878 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:41,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:41,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:41,878 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:41,878 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:41,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:41,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:41,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:41,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:41,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:41,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:41,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:41,880 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:41,880 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:41,880 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:41,880 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:41,880 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:41,881 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,881 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:41,881 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:41,881 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:41,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:41,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:41,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,882 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,885 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:41,885 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:41,885 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,885 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:41,885 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:41,885 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:41,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:41,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:41,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:41,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:41,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:41,885 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,885 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,885 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,885 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,885 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:41,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:41,885 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:41,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:41,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:41,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893061913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:41,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893061915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,985 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:41,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:41,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:41,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:41,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:41,988 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:41,988 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:41,988 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:41,988 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:41,989 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,989 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:41,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,990 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:41,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:41,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:41,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:41,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:41,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:41,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:41,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:41,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:41,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:41,991 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:41,991 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:41,991 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:41,991 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:41,991 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:41,991 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:41,991 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:41,991 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:41,992 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:41,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:41,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:41,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:41,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:41,992 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:41,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:41,993 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,993 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:41,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:41,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:41,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:41,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:41,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@785c0004[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:41,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,995 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:41,995 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:41,995 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:41,995 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:41,995 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,995 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:41,996 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:41,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:41,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:41,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:41,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:41,998 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:41,998 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:41,998 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:41,998 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:41,998 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:41,998 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,998 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:41,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:41,999 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:41,999 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,000 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:42,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,002 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:42,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:42,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:42,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:42,002 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:42,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:42,003 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:42,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:42,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:42,003 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,003 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,003 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:42,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:42,103 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:42,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:42,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:42,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:42,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:42,105 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:42,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:42,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:42,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:42,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,106 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:42,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:42,107 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,107 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:42,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:42,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:42,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:42,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:42,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:42,108 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:42,108 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:42,108 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:42,108 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:42,108 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:42,108 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:42,108 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:42,108 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:42,108 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,108 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:42,108 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:42,109 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,109 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,109 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,109 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:42,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:42,110 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:42,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a9d6c65[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:42,110 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,110 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:42,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:42,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:42,111 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,111 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:42,111 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:42,112 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:42,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:42,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:42,113 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:42,113 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:42,113 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:42,113 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:42,113 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:42,114 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,114 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,114 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:42,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,114 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:42,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:42,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,115 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,116 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:42,116 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:42,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:42,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:42,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:42,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:42,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,117 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:42,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,117 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:42,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:42,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:42,117 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:42,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:42,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:42,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893062115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:42,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893062117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,217 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:42,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:42,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:42,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:42,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:42,220 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:42,220 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:42,220 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:42,221 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:42,221 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303273f4c1f698a844b8b86f4150f6a3674f6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303273f4c1f698a844b8b86f4150f6a3674f6 2023-03-27 04:56:42,221 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,221 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,222 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,222 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:42,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:42,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:42,222 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,222 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:42,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:42,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,224 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:42,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:42,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:42,225 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:42,226 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:42,226 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:42,226 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:42,226 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:42,226 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:42,236 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:42,236 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:42,237 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:42,237 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:42,237 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:42,237 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,237 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:42,237 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:42,238 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,238 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,238 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,238 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:42,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:42,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@640825d7[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:42,240 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:42,240 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,241 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,241 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:42,241 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:42,241 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:42,242 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,242 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:42,243 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:42,243 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:42,243 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,243 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:42,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:42,246 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:42,246 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:42,246 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:42,246 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:42,246 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:42,247 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,247 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:42,247 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:42,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:42,248 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,249 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,253 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:42,253 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:42,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:42,253 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:42,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:42,253 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:42,253 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,254 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:42,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,254 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:42,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:42,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:42,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:42,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893062281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,285 DEBUG [Thread-987] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., hostname=jenkins-hbase20.apache.org,44923,1679892955505, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:56:42,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:42,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893062285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:42,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60422 deadline: 1679893062286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,290 DEBUG [Thread-979] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., hostname=jenkins-hbase20.apache.org,44923,1679892955505, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:56:42,293 DEBUG [Thread-983] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., hostname=jenkins-hbase20.apache.org,44923,1679892955505, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:56:42,354 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:42,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:42,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:42,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:42,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:42,357 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:42,357 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:42,357 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:42,357 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:42,357 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,357 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,358 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:42,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:42,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:42,358 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,358 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:42,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:42,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:42,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:42,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:42,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:42,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:42,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:42,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:42,360 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:42,360 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:42,360 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:42,360 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:42,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:42,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:42,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:42,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:42,362 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,362 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,362 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,362 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:42,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:42,363 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:42,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7df69666[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:42,363 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,364 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:42,364 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,364 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:42,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:42,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:42,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,365 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:42,365 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:42,365 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:42,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:42,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:42,367 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:42,367 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:42,367 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:42,367 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:42,367 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:42,368 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,368 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:42,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,368 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:42,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,369 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,372 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,374 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:42,374 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:42,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:42,374 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,374 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:42,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:42,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:42,374 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,374 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,374 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,374 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,374 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:42,374 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:42,374 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:42,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:42,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:42,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:42,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893062422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:42,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893062424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,474 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:42,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:42,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:42,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:42,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:42,477 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:42,477 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:42,477 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:42,477 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:42,478 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,478 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,478 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,478 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:42,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:42,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:42,479 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,479 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:42,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:42,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:42,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:42,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:42,480 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:42,480 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:42,480 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:42,480 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:42,480 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:42,480 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:42,480 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:42,480 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:42,480 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:42,481 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:42,481 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:42,481 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,481 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:42,481 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:42,481 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,481 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,482 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,482 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:42,483 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:42,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6532f5fc[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:42,483 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,483 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,483 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,484 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,484 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:42,484 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:42,484 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:42,484 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,484 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,484 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:42,485 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:42,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:42,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:42,486 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:42,486 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:42,486 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:42,486 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:42,486 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:42,487 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,487 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,487 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:42,488 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,488 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,490 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,490 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:42,490 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,490 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:42,490 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:42,490 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,490 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,490 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,490 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:42,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:42,491 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:42,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:42,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:42,492 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:42,492 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:42,592 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:42,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:42,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:42,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:42,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:42,595 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:42,595 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:42,595 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:42,595 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:42,595 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,595 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,596 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,596 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:42,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:42,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:42,596 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,596 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:42,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:42,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:42,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:42,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:42,597 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:42,597 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:42,597 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:42,597 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:42,597 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:42,597 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:42,598 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:42,598 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:42,598 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:42,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:42,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:42,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:42,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:42,599 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,599 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,599 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,599 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:42,600 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:42,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@446d437b[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:42,600 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,601 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:42,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:42,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:42,601 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,601 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:42,602 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,602 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:42,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,602 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:42,602 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:42,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:42,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:42,604 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:42,604 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:42,604 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:42,604 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:42,604 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:42,605 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,605 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,605 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:42,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:42,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:42,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,606 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,614 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,620 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:42,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:42,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:42,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:42,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:42,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:42,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:42,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:42,621 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:42,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:42,621 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:42,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:42,632 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=348, memsize=55.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/58f6022cbab9452681e688762d66633f 2023-03-27 04:56:42,721 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:42,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:42,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:42,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:42,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:42,724 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:42,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:42,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:42,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:42,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,725 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,725 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:42,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:42,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:42,725 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,725 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:42,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:42,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:42,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:42,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:42,726 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:42,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:42,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:42,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:42,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:42,727 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:42,727 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:42,727 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:42,727 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:42,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:42,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:42,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:42,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:42,737 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,737 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,737 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,737 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:42,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:42,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61eee0bc[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:42,739 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:42,739 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,739 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,739 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:42,739 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:42,739 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:42,739 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,739 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:42,741 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:42,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:42,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:42,743 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:42,743 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:42,743 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:42,743 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:42,743 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:42,744 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,744 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,744 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:42,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,744 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:42,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:42,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,753 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,754 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:42,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:42,755 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:42,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:42,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:42,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:42,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:42,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:42,755 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:42,755 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:42,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:42,855 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:42,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:42,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:42,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:42,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:42,858 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:42,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:42,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:42,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:42,859 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,859 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,860 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:42,860 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:42,860 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,860 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:42,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:42,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:42,863 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:42,863 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:42,863 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:42,863 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:42,863 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:42,863 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:42,863 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:42,863 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:42,863 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:42,863 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:42,863 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:42,863 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:42,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:42,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:42,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28e20f41[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:42,866 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:42,866 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,867 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,867 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:42,867 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:42,867 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:42,867 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,867 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:42,868 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:42,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:42,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:42,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:42,871 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:42,871 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:42,871 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:42,871 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:42,871 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:42,872 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,872 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,873 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,873 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:42,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,876 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,878 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:42,878 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:42,878 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,878 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,878 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,878 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,878 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,878 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,878 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,878 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:42,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:42,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:42,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:42,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:42,879 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:42,879 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:42,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:42,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:42,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:42,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:42,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893062927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:42,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893062928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,979 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:42,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:42,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:42,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:42,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:42,982 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:42,982 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:42,982 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:42,982 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:42,982 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,982 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,983 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,983 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:42,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:42,983 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,983 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:42,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:42,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:42,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:42,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:42,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:42,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:42,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:42,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:42,984 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:42,985 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:42,985 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:42,985 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:42,985 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:42,985 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:42,985 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,985 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:42,985 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:42,986 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:42,987 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:42,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a8d82b4[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:42,987 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,987 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:42,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:42,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:42,988 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,988 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:42,988 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:42,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:42,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:42,990 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:42,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:42,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:42,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:42,990 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:42,991 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,991 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:42,991 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:42,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:42,991 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,992 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:42,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,994 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,994 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,994 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:42,994 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:42,994 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:42,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:42,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:42,994 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:42,994 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,994 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:42,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:42,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:42,994 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:42,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:42,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:42,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:42,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:42,994 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:42,995 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:43,051 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/b54216b70e644931bb65a88a17c82d43 2023-03-27 04:56:43,095 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:43,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:43,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:43,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:43,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:43,097 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:43,097 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:43,097 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:43,097 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:43,098 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,098 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,099 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:43,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:43,099 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:43,099 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,099 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:43,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:43,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:43,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:43,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:43,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:43,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:43,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:43,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:43,101 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:43,101 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:43,101 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:43,101 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:43,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:43,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:43,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:43,102 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:43,102 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,102 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:43,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:43,103 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,103 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:43,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:43,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:43,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:43,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:43,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74557405[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:43,104 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:43,104 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,104 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,104 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:43,104 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:43,104 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:43,104 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,105 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:43,105 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:43,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:43,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:43,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:43,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:43,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:43,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:43,107 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:43,107 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:43,107 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:43,107 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:43,107 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:43,108 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,108 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:43,109 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:43,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:43,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:43,109 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,110 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,111 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,111 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,111 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,111 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,112 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:43,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:43,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:43,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:43,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:43,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:43,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,112 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:43,112 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:43,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:43,212 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:43,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:43,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:43,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:43,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:43,215 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:43,215 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:43,215 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:43,215 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:43,215 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,215 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,216 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,216 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:43,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:43,216 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,216 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:43,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:43,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:43,217 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:43,217 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:43,217 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:43,217 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:43,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:43,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:43,218 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:43,218 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:43,218 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:43,218 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:43,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:43,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:43,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:43,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:43,219 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,219 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:43,219 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,219 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:43,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:43,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:43,220 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:43,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62a89133[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:43,220 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,221 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,221 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:43,221 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:43,221 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:43,221 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,221 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:43,222 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:43,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:43,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:43,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:43,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:43,224 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:43,224 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:43,224 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:43,224 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:43,224 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:43,225 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,225 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,225 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:43,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:43,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:43,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:43,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:43,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:43,225 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,226 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:43,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:43,228 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:43,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:43,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:43,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:43,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:43,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,229 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:43,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:43,229 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:43,329 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:43,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:43,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:43,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:43,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:43,332 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:43,332 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:43,332 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:43,332 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:43,332 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,332 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,333 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:43,333 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:43,333 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,333 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:43,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:43,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:43,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:43,334 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:43,334 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:43,334 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:43,334 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:43,335 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:43,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:43,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:43,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:43,336 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:43,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a080220[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:43,336 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,337 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,337 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:43,337 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:43,337 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,337 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:43,337 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:43,338 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:43,338 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:43,338 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,338 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:43,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:43,340 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:43,340 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:43,340 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:43,340 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:43,340 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:43,341 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,341 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:43,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,341 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:43,341 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,342 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,343 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,343 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,343 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:43,343 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:43,343 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:43,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:43,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:43,343 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,344 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:43,344 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,344 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,344 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:43,344 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,344 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:43,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:43,444 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:43,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:43,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:43,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:43,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:43,447 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:43,447 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:43,447 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:43,447 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:43,447 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,447 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,448 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,448 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:43,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:43,448 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,448 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:43,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:43,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:43,449 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:43,449 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:43,449 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:43,449 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:43,449 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:43,449 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:43,449 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:43,450 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:43,450 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:43,451 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,451 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:43,452 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,452 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:43,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:43,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c26b3f7[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:43,453 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:43,453 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,454 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,454 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:43,454 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:43,454 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:43,454 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,454 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:43,455 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,455 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:43,455 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:43,455 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:43,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:43,457 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:43,457 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:43,457 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:43,457 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:43,457 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:43,458 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,458 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:43,458 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:43,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:43,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,459 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,460 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,463 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/ae6e633cca4e4d059a6a40ace66050a5 2023-03-27 04:56:43,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/58f6022cbab9452681e688762d66633f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/58f6022cbab9452681e688762d66633f 2023-03-27 04:56:43,473 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/58f6022cbab9452681e688762d66633f, entries=200, sequenceid=348, filesize=32.3 K 2023-03-27 04:56:43,474 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/b54216b70e644931bb65a88a17c82d43 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/b54216b70e644931bb65a88a17c82d43 2023-03-27 04:56:43,474 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,475 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:43,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:43,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:43,475 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,475 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:43,475 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,475 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:43,475 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,475 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:43,475 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:43,475 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,476 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:43,476 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:43,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:43,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:43,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:43,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:43,480 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/b54216b70e644931bb65a88a17c82d43, entries=150, sequenceid=348, filesize=11.9 K 2023-03-27 04:56:43,481 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/ae6e633cca4e4d059a6a40ace66050a5 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/ae6e633cca4e4d059a6a40ace66050a5 2023-03-27 04:56:43,497 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/ae6e633cca4e4d059a6a40ace66050a5, entries=150, sequenceid=348, filesize=11.9 K 2023-03-27 04:56:43,497 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~467.44 KB/478656, currentSize=20.13 KB/20610 for 13f5a923b0a0364586ae63fe46bf7293 in 1692ms, sequenceid=348, compaction requested=true 2023-03-27 04:56:43,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:43,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:43,498 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:43,498 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:43,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:43,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:43,499 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:43,499 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 13f5a923b0a0364586ae63fe46bf7293/A is initiating minor compaction (all files) 2023-03-27 04:56:43,499 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 13f5a923b0a0364586ae63fe46bf7293/A in TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:43,499 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/e4c2f0686a3a48639f989c12cf79e855, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0d4b69010b164bf28dd841228b8c40c1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/58f6022cbab9452681e688762d66633f] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp, totalSize=82.9 K 2023-03-27 04:56:43,499 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36246 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:43,499 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 13f5a923b0a0364586ae63fe46bf7293/B is initiating minor compaction (all files) 2023-03-27 04:56:43,500 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 13f5a923b0a0364586ae63fe46bf7293/B in TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:43,500 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/87c0a7193236450db9884f33e6f85d9b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/29cbccac02854bb0985a808e65c5fbc0, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/b54216b70e644931bb65a88a17c82d43] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp, totalSize=35.4 K 2023-03-27 04:56:43,500 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 87c0a7193236450db9884f33e6f85d9b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1679892998140 2023-03-27 04:56:43,500 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting e4c2f0686a3a48639f989c12cf79e855, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1679892998140 2023-03-27 04:56:43,501 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 0d4b69010b164bf28dd841228b8c40c1, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679892999506 2023-03-27 04:56:43,501 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 29cbccac02854bb0985a808e65c5fbc0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679892999506 2023-03-27 04:56:43,501 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 58f6022cbab9452681e688762d66633f, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1679893001192 2023-03-27 04:56:43,502 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting b54216b70e644931bb65a88a17c82d43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1679893001192 2023-03-27 04:56:43,510 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 13f5a923b0a0364586ae63fe46bf7293#B#compaction#147 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:43,514 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 13f5a923b0a0364586ae63fe46bf7293#A#compaction#146 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:43,520 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/d15a2f5100d749a697a5c53308057a83 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/d15a2f5100d749a697a5c53308057a83 2023-03-27 04:56:43,524 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 13f5a923b0a0364586ae63fe46bf7293/B of 13f5a923b0a0364586ae63fe46bf7293 into d15a2f5100d749a697a5c53308057a83(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:43,524 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:43,524 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., storeName=13f5a923b0a0364586ae63fe46bf7293/B, priority=13, startTime=1679893003498; duration=0sec 2023-03-27 04:56:43,524 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:43,525 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:43,525 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36246 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:43,526 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 13f5a923b0a0364586ae63fe46bf7293/C is initiating minor compaction (all files) 2023-03-27 04:56:43,526 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 13f5a923b0a0364586ae63fe46bf7293/C in TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:43,526 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/cb7f36d0a7a04284b6dc25e9ed62f221, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/a71d644a1c0b4bef976507b42b581b83, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/ae6e633cca4e4d059a6a40ace66050a5] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp, totalSize=35.4 K 2023-03-27 04:56:43,526 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting cb7f36d0a7a04284b6dc25e9ed62f221, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1679892998140 2023-03-27 04:56:43,526 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting a71d644a1c0b4bef976507b42b581b83, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679892999506 2023-03-27 04:56:43,527 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting ae6e633cca4e4d059a6a40ace66050a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1679893001192 2023-03-27 04:56:43,535 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 13f5a923b0a0364586ae63fe46bf7293#C#compaction#148 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:43,575 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:43,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:43,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:43,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:43,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:43,578 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:43,578 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:43,578 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:43,578 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:43,578 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,578 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,579 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,579 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:43,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:43,579 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:43,579 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:43,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:43,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:43,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:43,580 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:43,580 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:43,580 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:43,580 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:43,580 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:43,580 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:43,581 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:43,581 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-27 04:56:43,581 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:43,581 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:43,581 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:43,581 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:43,581 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:43,581 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:43,600 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303272d07b49cda174e97bc4f09d8362a0bcf to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303272d07b49cda174e97bc4f09d8362a0bcf 2023-03-27 04:56:43,611 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=355, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/18431024500d491ca45e367f7400d7b3 2023-03-27 04:56:43,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:43,980 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:43,980 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/fdacab010600457bbd664f6ba79ed5e9 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/fdacab010600457bbd664f6ba79ed5e9 2023-03-27 04:56:43,985 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 13f5a923b0a0364586ae63fe46bf7293/C of 13f5a923b0a0364586ae63fe46bf7293 into fdacab010600457bbd664f6ba79ed5e9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:43,985 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:43,985 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., storeName=13f5a923b0a0364586ae63fe46bf7293/C, priority=13, startTime=1679893003498; duration=0sec 2023-03-27 04:56:43,985 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:44,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893004009, max timestamp=1679893004022Num uniques 859; 2023-03-27 04:56:44,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:44,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893004009, max timestamp=1679893004022Num uniques 859; 2023-03-27 04:56:44,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893004009, max timestamp=1679893004022Num uniques 859; 2023-03-27 04:56:44,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:44,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:44,040 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/09b99d761bfa408691741d4ea358ca32 2023-03-27 04:56:44,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679893004039, max timestamp=1679893004049Num uniques 1068; 2023-03-27 04:56:44,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-27 04:56:44,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679893004039, max timestamp=1679893004049Num uniques 1068; 2023-03-27 04:56:44,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-27 04:56:44,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679893004039, max timestamp=1679893004049Num uniques 1068; 2023-03-27 04:56:44,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-27 04:56:44,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679893004055, max timestamp=1679893004074Num uniques 1127; 2023-03-27 04:56:44,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-03-27 04:56:44,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679893004068, max timestamp=1679893004074Num uniques 1127; 2023-03-27 04:56:44,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679893004068, max timestamp=1679893004074Num uniques 1127; 2023-03-27 04:56:44,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-27 04:56:44,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-27 04:56:44,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8509, totalHeapSize=19584, min timestamp=1679893004089, max timestamp=1679893004094Num uniques 1036; 2023-03-27 04:56:44,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679893004089, max timestamp=1679893004094Num uniques 1036; 2023-03-27 04:56:44,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47439, new segment data size=8509, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-27 04:56:44,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-27 04:56:44,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679893004089, max timestamp=1679893004094Num uniques 1186; 2023-03-27 04:56:44,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=8516, suffix heap size=151312,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=186 2023-03-27 04:56:44,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1679893004112, max timestamp=1679893004118Num uniques 1045; 2023-03-27 04:56:44,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-03-27 04:56:44,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1679893004112, max timestamp=1679893004118Num uniques 1045; 2023-03-27 04:56:44,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1679893004112, max timestamp=1679893004118Num uniques 1045; 2023-03-27 04:56:44,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-03-27 04:56:44,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-03-27 04:56:44,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:44,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679893004142, max timestamp=1679893004150Num uniques 1154; 2023-03-27 04:56:44,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893064149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-03-27 04:56:44,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679893004142, max timestamp=1679893004150Num uniques 1154; 2023-03-27 04:56:44,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-03-27 04:56:44,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679893004142, max timestamp=1679893004150Num uniques 1154; 2023-03-27 04:56:44,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-03-27 04:56:44,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679893004177, max timestamp=1679893004190Num uniques 1113; 2023-03-27 04:56:44,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-03-27 04:56:44,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679893004177, max timestamp=1679893004190Num uniques 1113; 2023-03-27 04:56:44,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-27 04:56:44,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679893004177, max timestamp=1679893004190Num uniques 1113; 2023-03-27 04:56:44,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-03-27 04:56:44,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679893004227, max timestamp=1679893004238Num uniques 1072; 2023-03-27 04:56:44,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-27 04:56:44,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679893004227, max timestamp=1679893004238Num uniques 1022; 2023-03-27 04:56:44,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-27 04:56:44,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679893004227, max timestamp=1679893004238Num uniques 1022; 2023-03-27 04:56:44,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-27 04:56:44,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679893004259, max timestamp=1679893004266Num uniques 1031; 2023-03-27 04:56:44,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679893004259, max timestamp=1679893004266Num uniques 1031; 2023-03-27 04:56:44,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-27 04:56:44,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679893004259, max timestamp=1679893004266Num uniques 1031; 2023-03-27 04:56:44,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-27 04:56:44,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-27 04:56:44,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1679893004274, max timestamp=1679893004288Num uniques 1090; 2023-03-27 04:56:44,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=15570, suffix heap size=141328,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=340 2023-03-27 04:56:44,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1679893004274, max timestamp=1679893004288Num uniques 1090; 2023-03-27 04:56:44,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=15570, suffix heap size=141328,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=340 2023-03-27 04:56:44,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1679893004274, max timestamp=1679893004288Num uniques 1090; 2023-03-27 04:56:44,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=15570, suffix heap size=141328,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=340 2023-03-27 04:56:44,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=399, cellSize=18274, totalHeapSize=41736, min timestamp=1679893004305, max timestamp=1679893004312Num uniques 1199; 2023-03-27 04:56:44,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=399, cellSize=18274, totalHeapSize=41736, min timestamp=1679893004305, max timestamp=1679893004312Num uniques 1199; 2023-03-27 04:56:44,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=18274, suffix heap size=152664,new segment heap size=41736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=399 2023-03-27 04:56:44,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=18274, suffix heap size=152664,new segment heap size=41736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=399 2023-03-27 04:56:44,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679893004305, max timestamp=1679893004312Num uniques 1199; 2023-03-27 04:56:44,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=9114, suffix heap size=152664,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=199 2023-03-27 04:56:44,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:44,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893064345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679893004342, max timestamp=1679893004345Num uniques 1258; 2023-03-27 04:56:44,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=57610, new segment data size=7230, suffix heap size=158800,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1258, new segment cells count=158 2023-03-27 04:56:44,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679893004342, max timestamp=1679893004345Num uniques 1258; 2023-03-27 04:56:44,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=57610, new segment data size=7230, suffix heap size=158800,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1258, new segment cells count=158 2023-03-27 04:56:44,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679893004342, max timestamp=1679893004345Num uniques 1058; 2023-03-27 04:56:44,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-27 04:56:44,384 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/bdbbd8ea598449a59b2ba28a75f9a190 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/bdbbd8ea598449a59b2ba28a75f9a190 2023-03-27 04:56:44,388 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 13f5a923b0a0364586ae63fe46bf7293/A of 13f5a923b0a0364586ae63fe46bf7293 into bdbbd8ea598449a59b2ba28a75f9a190(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:44,388 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:44,388 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., storeName=13f5a923b0a0364586ae63fe46bf7293/A, priority=13, startTime=1679893003498; duration=0sec 2023-03-27 04:56:44,389 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:44,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679893004376, max timestamp=1679893004399Num uniques 1017; 2023-03-27 04:56:44,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679893004376, max timestamp=1679893004399Num uniques 1017; 2023-03-27 04:56:44,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-03-27 04:56:44,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679893004376, max timestamp=1679893004399Num uniques 1017; 2023-03-27 04:56:44,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-03-27 04:56:44,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-03-27 04:56:44,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1679893004420, max timestamp=1679893004431Num uniques 1076; 2023-03-27 04:56:44,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-03-27 04:56:44,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1679893004420, max timestamp=1679893004431Num uniques 1076; 2023-03-27 04:56:44,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1679893004420, max timestamp=1679893004431Num uniques 1076; 2023-03-27 04:56:44,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-03-27 04:56:44,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-03-27 04:56:44,450 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/e7300ba3e1ef48a38559ed64bf55efa7 2023-03-27 04:56:44,457 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/18431024500d491ca45e367f7400d7b3 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/18431024500d491ca45e367f7400d7b3 2023-03-27 04:56:44,461 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/18431024500d491ca45e367f7400d7b3, entries=50, sequenceid=355, filesize=11.7 K 2023-03-27 04:56:44,462 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/09b99d761bfa408691741d4ea358ca32 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/09b99d761bfa408691741d4ea358ca32 2023-03-27 04:56:44,466 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/09b99d761bfa408691741d4ea358ca32, entries=50, sequenceid=355, filesize=7.1 K 2023-03-27 04:56:44,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679893004459, max timestamp=1679893004464Num uniques 1085; 2023-03-27 04:56:44,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-03-27 04:56:44,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679893004459, max timestamp=1679893004464Num uniques 1085; 2023-03-27 04:56:44,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679893004459, max timestamp=1679893004464Num uniques 1085; 2023-03-27 04:56:44,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-03-27 04:56:44,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-03-27 04:56:44,467 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/e7300ba3e1ef48a38559ed64bf55efa7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e7300ba3e1ef48a38559ed64bf55efa7 2023-03-27 04:56:44,471 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e7300ba3e1ef48a38559ed64bf55efa7, entries=50, sequenceid=355, filesize=7.1 K 2023-03-27 04:56:44,472 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=40.25 KB/41220 for 13f5a923b0a0364586ae63fe46bf7293 in 891ms, sequenceid=355, compaction requested=false 2023-03-27 04:56:44,472 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:44,472 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:44,472 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:44,472 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:44,472 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:44,473 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=53.67 KB heapSize=138.80 KB 2023-03-27 04:56:44,473 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:44,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:44,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:44,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:44,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:44,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:44,478 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:44,478 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:44,479 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,479 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:44,480 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,480 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:44,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:44,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:44,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:44,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:44,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@330c072a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:44,487 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:44,487 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,492 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,492 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:44,492 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:44,492 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:44,492 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,492 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:44,493 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,493 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:44,494 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:44,494 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:44,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:44,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:44,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:44,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,497 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327369d990ab29846899d5fcc83e9beba65 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327369d990ab29846899d5fcc83e9beba65 2023-03-27 04:56:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:44,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:44,498 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:44,498 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:44,498 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:44,498 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:44,498 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:44,500 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,500 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:44,500 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:44,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:44,501 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,504 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:44,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:44,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,506 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:44,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:44,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:44,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:44,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:44,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:44,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:44,506 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:44,506 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:44,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:44,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:44,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893004505, max timestamp=1679893004523Num uniques 859; 2023-03-27 04:56:44,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:44,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893004505, max timestamp=1679893004523Num uniques 859; 2023-03-27 04:56:44,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:44,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893004505, max timestamp=1679893004523Num uniques 859; 2023-03-27 04:56:44,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:44,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:44,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893064547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:44,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893064550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,606 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:44,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:44,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:44,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:44,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:44,609 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:44,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:44,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:44,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:44,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:44,610 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:44,610 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:44,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:44,610 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,610 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:44,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:44,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:44,611 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:44,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:44,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:44,611 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:44,611 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:44,611 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:44,612 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:44,612 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:44,612 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:44,612 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:44,612 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:44,612 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:44,612 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:44,612 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:44,612 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,612 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:44,612 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:44,613 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,613 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:44,613 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,613 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:44,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:44,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@150b2971[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:44,614 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:44,614 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,615 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,615 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:44,615 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:44,615 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:44,615 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,615 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:44,615 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,615 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:44,615 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:44,616 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:44,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:44,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:44,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:44,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:44,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:44,617 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:44,617 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:44,617 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:44,617 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:44,617 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:44,618 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,618 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:44,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,618 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:44,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:44,619 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:44,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,620 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:44,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,621 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,621 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:44,621 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:44,621 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:44,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:44,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:44,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:44,622 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:44,621 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:44,622 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,622 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,622 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:44,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:44,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:44,622 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:44,622 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,622 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:44,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:44,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:44,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893064651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:44,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893064655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,722 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:44,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:44,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:44,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:44,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:44,725 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:44,725 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:44,725 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:44,725 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:44,726 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,726 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:44,726 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:44,726 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:44,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:44,727 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:44,727 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:44,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:44,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:44,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:44,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:44,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:44,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:44,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:44,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:44,730 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:44,730 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:44,730 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:44,730 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:44,730 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:44,730 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:44,730 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:44,730 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:44,730 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,730 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:44,730 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:44,737 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,737 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:44,737 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,737 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:44,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:44,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:44,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f0830f2[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:44,738 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:44,738 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,742 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,742 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:44,742 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:44,742 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,742 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:44,742 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:44,743 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:44,743 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,743 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:44,743 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:44,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:44,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:44,745 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:44,745 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:44,745 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:44,745 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:44,745 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:44,746 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,746 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,746 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:44,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:44,746 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:44,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:44,749 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,759 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,759 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:44,759 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:44,759 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:44,759 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:44,759 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:44,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:44,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:44,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:44,759 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,760 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:44,760 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:44,760 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,760 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,760 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:44,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:44,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:44,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893064853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:44,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893064857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,860 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:44,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:44,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:44,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:44,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:44,862 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:44,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:44,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:44,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:44,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:44,864 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,864 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:44,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:44,864 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,864 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:44,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:44,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:44,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:44,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:44,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:44,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:44,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:44,877 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:44,878 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:44,878 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:44,878 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:44,878 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:44,878 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:44,878 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:44,878 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:44,878 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,878 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:44,878 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:44,887 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:44,887 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:44,887 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,887 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:44,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:44,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@566883ef[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:44,888 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:44,888 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,892 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,892 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:44,892 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:44,892 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:44,892 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,892 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:44,893 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:44,893 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,893 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:44,893 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:44,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:44,895 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:44,895 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:44,895 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:44,895 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:44,895 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:44,896 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,896 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:44,896 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:44,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:44,896 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,897 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,899 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,899 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:44,899 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:44,899 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:44,899 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:44,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:44,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:44,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:44,899 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:44,899 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:44,899 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,899 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:44,899 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:44,899 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:44,899 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:44,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:44,920 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=621, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/e42f68754f2a4746ab368d5256c07439 2023-03-27 04:56:44,999 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:45,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:45,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:45,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:45,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:45,002 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:45,002 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:45,002 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:45,002 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:45,002 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,002 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,003 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:45,003 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:45,003 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,003 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:45,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:45,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:45,004 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:45,004 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:45,004 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:45,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:45,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:45,006 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:45,006 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:45,006 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:45,006 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:45,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:45,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:45,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:45,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:45,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,017 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:45,017 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,017 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:45,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:45,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50efb5f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:45,018 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:45,018 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,020 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,020 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:45,020 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,020 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:45,020 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:45,020 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:45,021 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:45,021 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,021 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:45,021 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:45,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:45,023 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:45,023 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:45,023 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:45,023 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:45,023 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:45,024 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,024 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,024 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:45,024 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:45,025 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:45,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:45,027 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:45,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:45,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:45,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:45,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,027 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:45,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:45,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:45,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,027 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:45,128 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:45,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:45,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:45,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:45,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:45,131 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:45,131 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:45,131 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:45,131 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:45,131 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,131 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,132 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:45,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:45,132 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,132 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:45,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:45,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:45,133 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:45,133 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:45,133 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:45,133 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:45,133 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:45,133 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:45,133 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:45,133 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:45,133 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:45,133 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:45,133 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:45,133 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,134 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:45,134 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:45,134 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,134 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:45,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:45,135 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,135 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:45,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:45,136 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:45,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49020a2e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:45,136 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,136 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,136 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:45,136 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:45,136 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,136 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:45,136 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:45,137 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:45,138 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:45,138 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,138 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:45,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:45,139 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:45,139 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:45,139 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:45,139 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:45,139 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:45,140 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:45,140 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,140 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:45,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:45,140 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,141 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,153 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:45,153 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:45,153 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:45,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:45,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:45,153 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:45,153 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:45,154 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,154 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:45,154 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:45,154 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,154 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,154 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,154 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:45,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:45,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893065156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:45,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893065159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,254 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:45,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:45,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:45,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:45,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:45,256 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:45,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:45,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:45,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:45,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,258 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:45,258 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:45,258 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,258 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:45,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:45,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:45,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:45,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:45,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:45,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:45,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:45,259 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:45,259 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:45,259 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:45,259 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:45,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:45,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:45,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:45,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:45,261 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,261 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:45,261 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,261 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:45,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:45,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c33c5c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:45,262 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:45,263 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,263 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,263 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:45,263 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:45,263 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,263 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:45,263 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:45,264 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:45,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:45,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:45,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:45,266 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:45,266 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:45,266 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:45,266 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:45,266 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:45,267 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,267 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:45,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,267 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:45,268 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,269 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,270 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,270 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,270 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,270 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:45,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:45,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:45,270 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:45,270 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:45,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:45,270 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:45,271 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,271 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:45,271 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:45,271 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:45,271 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:45,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:45,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=621 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/3b4517319f034ae39ed142cef9c68a91 2023-03-27 04:56:45,339 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=621 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/87e6348612304f31b771404f66793685 2023-03-27 04:56:45,347 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/e42f68754f2a4746ab368d5256c07439 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/e42f68754f2a4746ab368d5256c07439 2023-03-27 04:56:45,350 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/e42f68754f2a4746ab368d5256c07439, entries=250, sequenceid=621, filesize=39.2 K 2023-03-27 04:56:45,351 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/3b4517319f034ae39ed142cef9c68a91 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/3b4517319f034ae39ed142cef9c68a91 2023-03-27 04:56:45,356 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/3b4517319f034ae39ed142cef9c68a91, entries=150, sequenceid=621, filesize=11.9 K 2023-03-27 04:56:45,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/87e6348612304f31b771404f66793685 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/87e6348612304f31b771404f66793685 2023-03-27 04:56:45,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/87e6348612304f31b771404f66793685, entries=150, sequenceid=621, filesize=11.9 K 2023-03-27 04:56:45,362 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~138.75 KB/142080, currentSize=134.18 KB/137400 for 13f5a923b0a0364586ae63fe46bf7293 in 889ms, sequenceid=621, compaction requested=true 2023-03-27 04:56:45,362 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:45,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:45,362 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:45,362 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:45,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:45,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:45,363 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 31912 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:45,363 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78372 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:45,363 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 13f5a923b0a0364586ae63fe46bf7293/B is initiating minor compaction (all files) 2023-03-27 04:56:45,363 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 13f5a923b0a0364586ae63fe46bf7293/A is initiating minor compaction (all files) 2023-03-27 04:56:45,363 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 13f5a923b0a0364586ae63fe46bf7293/B in TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:45,363 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 13f5a923b0a0364586ae63fe46bf7293/A in TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:45,363 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/bdbbd8ea598449a59b2ba28a75f9a190, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/18431024500d491ca45e367f7400d7b3, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/e42f68754f2a4746ab368d5256c07439] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp, totalSize=76.5 K 2023-03-27 04:56:45,363 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/d15a2f5100d749a697a5c53308057a83, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/09b99d761bfa408691741d4ea358ca32, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/3b4517319f034ae39ed142cef9c68a91] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp, totalSize=31.2 K 2023-03-27 04:56:45,364 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting bdbbd8ea598449a59b2ba28a75f9a190, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1679893001192 2023-03-27 04:56:45,364 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting d15a2f5100d749a697a5c53308057a83, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1679893001192 2023-03-27 04:56:45,364 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 18431024500d491ca45e367f7400d7b3, keycount=50, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1679893001811 2023-03-27 04:56:45,364 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 09b99d761bfa408691741d4ea358ca32, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1679893001811 2023-03-27 04:56:45,365 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting e42f68754f2a4746ab368d5256c07439, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1679893004460 2023-03-27 04:56:45,365 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 3b4517319f034ae39ed142cef9c68a91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1679893004464 2023-03-27 04:56:45,371 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:45,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:45,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:45,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:45,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:45,378 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:45,378 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:45,378 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:45,378 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:45,379 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,379 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,380 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:45,380 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:45,381 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,381 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:45,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:45,382 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:45,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:45,382 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:45,382 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:45,382 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:45,383 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 13f5a923b0a0364586ae63fe46bf7293#B#compaction#156 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:45,389 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:45,389 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:45,389 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 13f5a923b0a0364586ae63fe46bf7293#A#compaction#155 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:45,390 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:45,390 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:45,390 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-27 04:56:45,391 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:45,391 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:45,391 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:45,391 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:45,391 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:45,391 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:45,426 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327a0b333ea160f44fdacc6b74756f71801 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327a0b333ea160f44fdacc6b74756f71801 2023-03-27 04:56:45,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:45,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:45,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:45,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893065670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:45,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893065670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:45,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893065771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:45,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893065771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,810 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/6f60b67e068d414d973c522391cbce9f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/6f60b67e068d414d973c522391cbce9f 2023-03-27 04:56:45,815 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 13f5a923b0a0364586ae63fe46bf7293/B of 13f5a923b0a0364586ae63fe46bf7293 into 6f60b67e068d414d973c522391cbce9f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:45,815 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:45,815 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., storeName=13f5a923b0a0364586ae63fe46bf7293/B, priority=13, startTime=1679893005362; duration=0sec 2023-03-27 04:56:45,815 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:45,815 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:45,816 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 31912 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:45,816 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 13f5a923b0a0364586ae63fe46bf7293/C is initiating minor compaction (all files) 2023-03-27 04:56:45,816 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 13f5a923b0a0364586ae63fe46bf7293/C in TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:45,816 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/fdacab010600457bbd664f6ba79ed5e9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e7300ba3e1ef48a38559ed64bf55efa7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/87e6348612304f31b771404f66793685] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp, totalSize=31.2 K 2023-03-27 04:56:45,817 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting fdacab010600457bbd664f6ba79ed5e9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1679893001192 2023-03-27 04:56:45,817 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting e7300ba3e1ef48a38559ed64bf55efa7, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1679893001811 2023-03-27 04:56:45,817 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 87e6348612304f31b771404f66793685, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1679893004464 2023-03-27 04:56:45,831 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 13f5a923b0a0364586ae63fe46bf7293#C#compaction#158 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:45,832 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=657, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/0d15f62738234de8b42053b759074801 2023-03-27 04:56:45,839 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/af08e79bcb6541299482ac29d930b407 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/af08e79bcb6541299482ac29d930b407 2023-03-27 04:56:45,844 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 13f5a923b0a0364586ae63fe46bf7293/A of 13f5a923b0a0364586ae63fe46bf7293 into af08e79bcb6541299482ac29d930b407(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:45,844 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:45,844 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., storeName=13f5a923b0a0364586ae63fe46bf7293/A, priority=13, startTime=1679893005362; duration=0sec 2023-03-27 04:56:45,844 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:45,848 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/6e7231408c804f23bad9baeaab471bad as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/6e7231408c804f23bad9baeaab471bad 2023-03-27 04:56:45,850 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/ee7e717125fa448ab5caf0daa8c88cfb 2023-03-27 04:56:45,856 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 13f5a923b0a0364586ae63fe46bf7293/C of 13f5a923b0a0364586ae63fe46bf7293 into 6e7231408c804f23bad9baeaab471bad(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:45,856 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:45,856 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., storeName=13f5a923b0a0364586ae63fe46bf7293/C, priority=13, startTime=1679893005362; duration=0sec 2023-03-27 04:56:45,856 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:45,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:45,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893065976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:45,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:45,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893065976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,265 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/3a198304f3244b79af6fb4ad42312bda 2023-03-27 04:56:46,270 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/0d15f62738234de8b42053b759074801 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0d15f62738234de8b42053b759074801 2023-03-27 04:56:46,273 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0d15f62738234de8b42053b759074801, entries=150, sequenceid=657, filesize=25.4 K 2023-03-27 04:56:46,275 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/ee7e717125fa448ab5caf0daa8c88cfb as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/ee7e717125fa448ab5caf0daa8c88cfb 2023-03-27 04:56:46,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:46,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893066278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:46,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893066278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,281 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/ee7e717125fa448ab5caf0daa8c88cfb, entries=150, sequenceid=657, filesize=11.9 K 2023-03-27 04:56:46,283 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/3a198304f3244b79af6fb4ad42312bda as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/3a198304f3244b79af6fb4ad42312bda 2023-03-27 04:56:46,287 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/3a198304f3244b79af6fb4ad42312bda, entries=150, sequenceid=657, filesize=11.9 K 2023-03-27 04:56:46,288 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 13f5a923b0a0364586ae63fe46bf7293 in 898ms, sequenceid=657, compaction requested=false 2023-03-27 04:56:46,288 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:46,288 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:46,288 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:46,288 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:46,288 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,288 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:46,288 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:46,289 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,289 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,289 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,289 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:46,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1526a0b0[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:46,291 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:46,291 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,291 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,291 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:46,291 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:46,291 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:46,292 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:46,292 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:46,292 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:46,292 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,292 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:46,292 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:46,292 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:46,293 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:46,293 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:46,293 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:46,293 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:46,293 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,294 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,294 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,294 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:46,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,295 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,296 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:46,296 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:46,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:46,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:46,296 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,297 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,297 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,297 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,297 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,297 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,297 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:46,297 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:46,297 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:46,298 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:46,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:46,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:46,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:46,300 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-27 04:56:46,300 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:46,301 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:46,301 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:46,301 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:46,301 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:46,301 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:46,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:46,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893066324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:46,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893066324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:46,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60422 deadline: 1679893066332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,334 DEBUG [Thread-983] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8183 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., hostname=jenkins-hbase20.apache.org,44923,1679892955505, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:56:46,398 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:46,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:46,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:46,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:46,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:46,400 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:46,400 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:46,400 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:46,400 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:46,401 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,401 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,401 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,401 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:46,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:46,402 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,402 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:46,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:46,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:46,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:46,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:46,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:46,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:46,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:46,403 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:46,403 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:46,403 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:46,403 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:46,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:46,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:46,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:46,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:46,404 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,404 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,404 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,404 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:46,405 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:46,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67007548[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:46,405 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,406 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,406 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:46,406 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:46,406 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:46,406 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,406 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:46,407 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,407 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:46,407 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,407 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:46,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:46,409 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:46,409 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:46,409 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:46,409 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:46,409 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:46,410 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,410 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,410 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:46,410 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,411 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,412 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,412 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,412 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:46,412 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:46,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:46,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:46,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:46,413 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:46,412 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:46,413 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,413 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,413 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:46,413 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:46,413 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:46,413 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:46,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:46,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893066426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:46,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893066426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,513 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:46,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:46,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:46,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:46,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:46,516 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:46,516 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:46,516 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:46,516 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:46,516 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,516 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,517 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,517 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:46,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:46,517 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,517 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:46,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:46,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:46,519 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:46,519 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:46,519 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:46,519 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:46,519 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:46,519 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:46,519 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:46,519 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:46,519 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:46,519 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:46,519 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:46,519 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,519 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:46,519 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:46,520 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,520 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,520 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,520 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:46,521 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:46,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e8a40fc[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:46,521 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,523 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,523 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:46,523 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:46,523 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,523 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:46,523 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:46,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:46,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:46,524 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:46,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:46,526 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:46,526 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:46,526 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:46,526 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:46,526 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:46,527 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,527 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,527 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:46,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:46,527 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,528 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,530 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,530 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:46,530 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:46,530 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:46,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:46,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:46,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:46,530 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:46,530 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,530 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,530 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,530 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:46,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:46,531 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:46,530 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,531 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:46,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893066629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,631 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:46,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:46,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893066629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:46,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:46,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:46,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:46,633 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:46,633 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:46,633 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:46,633 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:46,634 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,634 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,634 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:46,634 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:46,634 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,635 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:46,635 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:46,635 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:46,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:46,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:46,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:46,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:46,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:46,636 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:46,636 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:46,636 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:46,636 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:46,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:46,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:46,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:46,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:46,637 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,637 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,637 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,637 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:46,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2035baad[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:46,638 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:46,638 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,639 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,639 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:46,639 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:46,639 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:46,639 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,639 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:46,640 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,640 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,640 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:46,640 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:46,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:46,641 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:46,642 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:46,642 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:46,642 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:46,642 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:46,642 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,642 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,643 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:46,643 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,644 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:46,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:46,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:46,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:46,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,645 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:46,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:46,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:46,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,646 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,646 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,646 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,646 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:46,646 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:46,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:46,646 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:46,725 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327c4b1eaecd6d043b99d64447f21433b2c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327c4b1eaecd6d043b99d64447f21433b2c 2023-03-27 04:56:46,746 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:46,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:46,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:46,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:46,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:46,749 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:46,749 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:46,749 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:46,749 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:46,749 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,749 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,750 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:46,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:46,750 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,750 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:46,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:46,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:46,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:46,751 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:46,751 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:46,751 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:46,751 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:46,751 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:46,751 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:46,751 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:46,751 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:46,752 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:46,752 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:46,752 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:46,752 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,752 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:46,752 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:46,752 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,752 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,752 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,753 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:46,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ad3b[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:46,754 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:46,754 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,755 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:46,755 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:46,755 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,755 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:46,755 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:46,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:46,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:46,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:46,756 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:46,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:46,760 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:46,760 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:46,760 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:46,760 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:46,760 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:46,761 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,761 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,761 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:46,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,761 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,762 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,764 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:46,764 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:46,764 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:46,764 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:46,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:46,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:46,764 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,764 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,764 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,764 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,764 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:46,764 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:46,764 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:46,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:46,764 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:46,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:46,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893066779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:46,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893066782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,864 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:46,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:46,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:46,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:46,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:46,867 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:46,867 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:46,867 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:46,867 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:46,867 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,867 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,868 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:46,868 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:46,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:46,868 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,868 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:46,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:46,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:46,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:46,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:46,869 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:46,869 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:46,869 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:46,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:46,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:46,870 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:46,870 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:46,870 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:46,870 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:46,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:46,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:46,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:46,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:46,871 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,871 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,871 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,871 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:46,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:46,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:46,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64782240[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:46,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,873 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,873 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:46,873 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:46,873 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:46,873 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,873 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:46,874 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:46,875 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:46,875 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:46,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:46,876 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:46,876 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:46,876 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:46,876 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:46,876 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:46,877 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,877 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,877 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:46,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,878 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:46,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:46,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,879 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,880 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,881 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,881 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,881 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,881 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:46,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:46,881 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,881 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:46,881 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:46,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:46,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:46,881 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:46,881 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:46,881 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:46,881 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:46,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:46,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:46,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:46,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893066933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:46,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893066933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,981 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:46,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:46,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:46,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:46,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:46,984 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:46,984 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:46,984 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:46,984 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:46,984 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,984 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,985 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:46,985 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:46,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,986 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,986 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:46,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:46,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:46,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:46,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:46,987 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:46,987 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:46,987 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:46,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:46,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:46,988 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:46,988 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:46,988 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:46,988 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:46,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:46,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:46,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:46,988 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:46,989 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,989 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,989 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,989 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:46,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8990393[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:46,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:46,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,990 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,991 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:46,991 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:46,991 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:46,991 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,991 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:46,991 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:46,991 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,991 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,991 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:46,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:46,993 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:46,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:46,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:46,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:46,993 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:46,994 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,994 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,994 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:46,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,995 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,997 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,997 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,997 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:46,997 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:46,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:46,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:46,997 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:46,997 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:46,997 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,997 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:46,997 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:46,997 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:46,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:46,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:46,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:46,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:46,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:46,998 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:46,998 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:47,098 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:47,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:47,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:47,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:47,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:47,101 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:47,101 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:47,101 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:47,101 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:47,101 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,101 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,102 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:47,102 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:47,102 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,102 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:47,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:47,103 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:47,103 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:47,103 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:47,103 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:47,103 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:47,103 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,103 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:47,103 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:47,103 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,104 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:47,104 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:47,104 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,104 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:47,104 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:47,104 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,104 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:47,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:47,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57430ff[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:47,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,106 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:47,106 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:47,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:47,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:47,107 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:47,108 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:47,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:47,109 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:47,109 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:47,109 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:47,109 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:47,109 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:47,110 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,110 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,110 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,110 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,112 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,113 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,113 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:47,113 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:47,113 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:47,113 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:47,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:47,113 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,113 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,114 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,114 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:47,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:47,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:47,114 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,114 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:47,114 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:47,132 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=672, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/4273d343f5ae46f9967cbba4acc0d455 2023-03-27 04:56:47,141 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/8953d36b53bf4aff98a98808ee194104 2023-03-27 04:56:47,150 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/e7662f89656944d091a2678b93dde3a3 2023-03-27 04:56:47,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/4273d343f5ae46f9967cbba4acc0d455 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/4273d343f5ae46f9967cbba4acc0d455 2023-03-27 04:56:47,158 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/4273d343f5ae46f9967cbba4acc0d455, entries=250, sequenceid=672, filesize=39.2 K 2023-03-27 04:56:47,159 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/8953d36b53bf4aff98a98808ee194104 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/8953d36b53bf4aff98a98808ee194104 2023-03-27 04:56:47,162 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/8953d36b53bf4aff98a98808ee194104, entries=150, sequenceid=672, filesize=11.9 K 2023-03-27 04:56:47,163 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/e7662f89656944d091a2678b93dde3a3 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e7662f89656944d091a2678b93dde3a3 2023-03-27 04:56:47,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e7662f89656944d091a2678b93dde3a3, entries=150, sequenceid=672, filesize=11.9 K 2023-03-27 04:56:47,169 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 13f5a923b0a0364586ae63fe46bf7293 in 869ms, sequenceid=672, compaction requested=true 2023-03-27 04:56:47,169 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:47,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:47,169 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:47,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:47,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:47,169 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:47,170 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92564 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:47,170 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:47,171 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 13f5a923b0a0364586ae63fe46bf7293/A is initiating minor compaction (all files) 2023-03-27 04:56:47,171 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 13f5a923b0a0364586ae63fe46bf7293/B is initiating minor compaction (all files) 2023-03-27 04:56:47,171 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 13f5a923b0a0364586ae63fe46bf7293/A in TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,171 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 13f5a923b0a0364586ae63fe46bf7293/B in TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,171 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/af08e79bcb6541299482ac29d930b407, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0d15f62738234de8b42053b759074801, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/4273d343f5ae46f9967cbba4acc0d455] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp, totalSize=90.4 K 2023-03-27 04:56:47,171 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/6f60b67e068d414d973c522391cbce9f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/ee7e717125fa448ab5caf0daa8c88cfb, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/8953d36b53bf4aff98a98808ee194104] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp, totalSize=36.0 K 2023-03-27 04:56:47,171 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting af08e79bcb6541299482ac29d930b407, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1679893004464 2023-03-27 04:56:47,171 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 6f60b67e068d414d973c522391cbce9f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1679893004464 2023-03-27 04:56:47,171 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 0d15f62738234de8b42053b759074801, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1679893004536 2023-03-27 04:56:47,171 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting ee7e717125fa448ab5caf0daa8c88cfb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1679893004536 2023-03-27 04:56:47,172 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 4273d343f5ae46f9967cbba4acc0d455, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1679893005665 2023-03-27 04:56:47,172 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 8953d36b53bf4aff98a98808ee194104, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1679893005669 2023-03-27 04:56:47,187 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 13f5a923b0a0364586ae63fe46bf7293#B#compaction#164 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:47,197 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 13f5a923b0a0364586ae63fe46bf7293#A#compaction#165 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:47,201 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/8d2c9c00df1f4d1ca61b9e1a40958712 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/8d2c9c00df1f4d1ca61b9e1a40958712 2023-03-27 04:56:47,209 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 13f5a923b0a0364586ae63fe46bf7293/B of 13f5a923b0a0364586ae63fe46bf7293 into 8d2c9c00df1f4d1ca61b9e1a40958712(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:47,209 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:47,209 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., storeName=13f5a923b0a0364586ae63fe46bf7293/B, priority=13, startTime=1679893007169; duration=0sec 2023-03-27 04:56:47,209 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:47,209 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:47,211 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:47,211 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 13f5a923b0a0364586ae63fe46bf7293/C is initiating minor compaction (all files) 2023-03-27 04:56:47,211 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 13f5a923b0a0364586ae63fe46bf7293/C in TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,211 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/6e7231408c804f23bad9baeaab471bad, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/3a198304f3244b79af6fb4ad42312bda, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e7662f89656944d091a2678b93dde3a3] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp, totalSize=36.0 K 2023-03-27 04:56:47,211 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 6e7231408c804f23bad9baeaab471bad, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1679893004464 2023-03-27 04:56:47,212 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 3a198304f3244b79af6fb4ad42312bda, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1679893004536 2023-03-27 04:56:47,212 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting e7662f89656944d091a2678b93dde3a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1679893005669 2023-03-27 04:56:47,214 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:47,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:47,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:47,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:47,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:47,224 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:47,224 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:47,224 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:47,224 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:47,224 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,224 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,225 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:47,225 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 13f5a923b0a0364586ae63fe46bf7293#C#compaction#166 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:47,225 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:47,225 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,225 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:47,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:47,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:47,226 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:47,226 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:47,226 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:47,227 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:47,227 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:47,227 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,227 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:47,227 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-27 04:56:47,227 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:47,228 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:47,228 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:47,228 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:47,228 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:47,228 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:47,242 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327cc7368eabc894b709b286a269685cf82 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327cc7368eabc894b709b286a269685cf82 2023-03-27 04:56:47,249 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=693, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/513c298683f944d18794e9ba9a24350f 2023-03-27 04:56:47,259 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=693 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/2d57901949814c59b1b283ddd5416049 2023-03-27 04:56:47,270 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=693 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/f84d56c32b004163a1fb5d4bbc9e9c35 2023-03-27 04:56:47,275 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/513c298683f944d18794e9ba9a24350f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/513c298683f944d18794e9ba9a24350f 2023-03-27 04:56:47,278 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/513c298683f944d18794e9ba9a24350f, entries=150, sequenceid=693, filesize=25.4 K 2023-03-27 04:56:47,279 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/2d57901949814c59b1b283ddd5416049 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/2d57901949814c59b1b283ddd5416049 2023-03-27 04:56:47,282 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/2d57901949814c59b1b283ddd5416049, entries=150, sequenceid=693, filesize=11.9 K 2023-03-27 04:56:47,283 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/f84d56c32b004163a1fb5d4bbc9e9c35 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/f84d56c32b004163a1fb5d4bbc9e9c35 2023-03-27 04:56:47,287 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/f84d56c32b004163a1fb5d4bbc9e9c35, entries=150, sequenceid=693, filesize=11.9 K 2023-03-27 04:56:47,288 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 13f5a923b0a0364586ae63fe46bf7293 in 61ms, sequenceid=693, compaction requested=false 2023-03-27 04:56:47,288 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:47,288 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,288 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:47,288 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:47,288 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,290 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:47,290 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:47,291 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,291 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,293 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,293 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:47,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:47,298 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:47,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@301e1a1f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:47,298 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,303 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,304 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:47,304 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:47,304 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:47,304 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,304 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:47,308 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,308 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,308 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:47,309 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:47,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:47,311 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:47,311 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:47,311 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:47,311 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:47,311 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:47,312 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,312 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:47,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,312 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:47,312 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,313 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,315 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:47,315 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:47,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:47,316 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,316 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,316 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:47,316 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,316 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,316 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,316 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,316 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,316 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:47,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:47,316 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:47,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:47,416 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:47,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:47,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:47,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:47,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:47,420 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:47,420 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:47,420 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:47,420 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:47,421 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,421 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,421 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:47,421 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:47,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:47,422 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,422 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:47,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:47,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:47,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:47,424 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:47,425 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:47,425 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:47,425 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:47,425 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:47,425 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,425 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:47,425 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:47,425 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,425 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:47,425 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:47,425 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,425 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:47,425 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:47,426 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,426 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:47,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:47,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15d1547e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:47,428 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:47,428 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,429 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:47,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:47,429 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,429 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:47,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:47,430 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,430 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:47,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,430 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:47,430 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:47,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:47,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:47,433 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:47,433 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:47,433 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:47,433 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:47,433 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:47,434 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,434 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:47,434 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,435 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,436 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,438 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:47,438 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:47,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:47,438 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,438 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,438 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,438 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,438 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,438 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,438 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,438 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:47,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:47,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:47,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:47,439 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:47,439 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:47,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:47,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:47,465 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:56:47,466 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:47,466 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:47,466 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:47,466 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:47,466 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:47,466 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:47,497 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327450b20a5296c4364a81a32c851cc1418 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327450b20a5296c4364a81a32c851cc1418 2023-03-27 04:56:47,508 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=704, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/2e7651113b0b483db6422021e952177a 2023-03-27 04:56:47,529 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=704 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/72ca5eb9bd15444da6c83cee3c3f5af4 2023-03-27 04:56:47,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893007514, max timestamp=1679893007534Num uniques 859; 2023-03-27 04:56:47,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893007514, max timestamp=1679893007534Num uniques 859; 2023-03-27 04:56:47,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893007514, max timestamp=1679893007534Num uniques 859; 2023-03-27 04:56:47,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:47,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:47,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:47,539 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:47,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:47,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:47,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:47,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:47,550 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:47,550 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:47,550 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:47,550 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:47,551 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,551 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,551 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:47,551 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:47,552 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,552 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:47,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:47,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:47,553 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:47,553 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:47,554 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:47,554 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:47,554 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:47,554 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,554 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:47,554 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:47,554 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,554 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:47,554 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:47,554 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,557 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:47,557 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:47,566 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=704 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/0178dc476ff74650ac96910ed0b92590 2023-03-27 04:56:47,567 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,567 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,567 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,567 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:47,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:47,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559458cc[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:47,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:47,569 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:47,569 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893067563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,570 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,570 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:47,570 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,571 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:47,571 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:47,571 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:47,571 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/2e7651113b0b483db6422021e952177a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/2e7651113b0b483db6422021e952177a 2023-03-27 04:56:47,571 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:47,571 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:47,572 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:47,572 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:47,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893067569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:47,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:47,575 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:47,575 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:47,575 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:47,575 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:47,575 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:47,578 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,578 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,579 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,580 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/2e7651113b0b483db6422021e952177a, entries=350, sequenceid=704, filesize=52.9 K 2023-03-27 04:56:47,582 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/72ca5eb9bd15444da6c83cee3c3f5af4 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/72ca5eb9bd15444da6c83cee3c3f5af4 2023-03-27 04:56:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,585 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,586 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/72ca5eb9bd15444da6c83cee3c3f5af4, entries=350, sequenceid=704, filesize=21.4 K 2023-03-27 04:56:47,587 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:47,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:47,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:47,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:47,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:47,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:47,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,587 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:47,587 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:47,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:47,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:47,588 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/0178dc476ff74650ac96910ed0b92590 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/0178dc476ff74650ac96910ed0b92590 2023-03-27 04:56:47,592 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/0178dc476ff74650ac96910ed0b92590, entries=150, sequenceid=704, filesize=11.9 K 2023-03-27 04:56:47,592 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 13f5a923b0a0364586ae63fe46bf7293 in 127ms, sequenceid=704, compaction requested=true 2023-03-27 04:56:47,592 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:47,592 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:47,593 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-27 04:56:47,593 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-27 04:56:47,633 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/0ae15e9c1b5747a3a40261f274c31952 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/0ae15e9c1b5747a3a40261f274c31952 2023-03-27 04:56:47,638 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 13f5a923b0a0364586ae63fe46bf7293/C of 13f5a923b0a0364586ae63fe46bf7293 into 0ae15e9c1b5747a3a40261f274c31952(size=12.3 K), total size for store is 36.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:47,638 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:47,638 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., storeName=13f5a923b0a0364586ae63fe46bf7293/C, priority=13, startTime=1679893007169; duration=0sec 2023-03-27 04:56:47,638 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-27 04:56:47,638 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-03-27 04:56:47,639 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:47,639 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:47,639 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. because compaction request was cancelled 2023-03-27 04:56:47,639 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:47,640 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 46766 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:47,640 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 13f5a923b0a0364586ae63fe46bf7293/B is initiating minor compaction (all files) 2023-03-27 04:56:47,641 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 13f5a923b0a0364586ae63fe46bf7293/B in TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,641 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/8d2c9c00df1f4d1ca61b9e1a40958712, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/2d57901949814c59b1b283ddd5416049, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/72ca5eb9bd15444da6c83cee3c3f5af4] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp, totalSize=45.7 K 2023-03-27 04:56:47,641 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 8d2c9c00df1f4d1ca61b9e1a40958712, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1679893005669 2023-03-27 04:56:47,641 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 2d57901949814c59b1b283ddd5416049, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=693, earliestPutTs=1679893006318 2023-03-27 04:56:47,642 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 72ca5eb9bd15444da6c83cee3c3f5af4, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=704, earliestPutTs=1679893007448 2023-03-27 04:56:47,659 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 13f5a923b0a0364586ae63fe46bf7293#B#compaction#173 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:47,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:47,675 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-27 04:56:47,675 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:47,675 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:47,675 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:47,676 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:47,676 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:47,676 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:47,685 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/52e2096c0fa447f2a1362debf6681fab as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/52e2096c0fa447f2a1362debf6681fab 2023-03-27 04:56:47,687 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:47,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:47,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:47,695 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 13f5a923b0a0364586ae63fe46bf7293/B of 13f5a923b0a0364586ae63fe46bf7293 into 52e2096c0fa447f2a1362debf6681fab(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:47,695 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:47,695 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., storeName=13f5a923b0a0364586ae63fe46bf7293/B, priority=13, startTime=1679893007593; duration=0sec 2023-03-27 04:56:47,695 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:47,695 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:47,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:47,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893067694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,698 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:47,698 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 13f5a923b0a0364586ae63fe46bf7293/C is initiating minor compaction (all files) 2023-03-27 04:56:47,699 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 13f5a923b0a0364586ae63fe46bf7293/C in TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,699 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/0ae15e9c1b5747a3a40261f274c31952, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/f84d56c32b004163a1fb5d4bbc9e9c35, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/0178dc476ff74650ac96910ed0b92590] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp, totalSize=36.1 K 2023-03-27 04:56:47,699 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 0ae15e9c1b5747a3a40261f274c31952, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1679893005669 2023-03-27 04:56:47,700 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting f84d56c32b004163a1fb5d4bbc9e9c35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=693, earliestPutTs=1679893006318 2023-03-27 04:56:47,700 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 0178dc476ff74650ac96910ed0b92590, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=704, earliestPutTs=1679893007454 2023-03-27 04:56:47,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:47,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893067695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:47,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:47,714 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:47,714 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:47,714 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:47,714 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:47,715 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,715 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,716 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,716 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:47,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:47,716 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,716 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,717 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:47,728 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 13f5a923b0a0364586ae63fe46bf7293#C#compaction#175 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:47,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:47,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:47,734 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:47,734 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:47,734 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:47,737 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:47,737 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:47,738 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,738 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:47,738 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:47,738 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327d80399e775c747b2b652373bbf6f600c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327d80399e775c747b2b652373bbf6f600c 2023-03-27 04:56:47,738 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,738 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:47,738 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:47,738 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,742 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:47,742 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:47,743 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,743 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,743 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,743 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:47,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71c36949[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:47,745 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:47,745 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,746 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,746 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:47,746 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:47,746 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:47,746 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,746 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:47,747 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:47,747 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:47,747 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,747 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:47,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:47,749 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:47,749 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:47,749 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:47,749 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:47,749 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:47,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:47,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:47,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:47,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,751 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,752 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,754 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,755 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:47,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,755 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:47,755 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,756 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:47,756 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:47,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:47,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:47,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:47,777 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=742, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/c3a75da504d24408972c73ad6944a503 2023-03-27 04:56:47,785 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/41c55ffbf6154d778a2be422cf4c5a25 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/41c55ffbf6154d778a2be422cf4c5a25 2023-03-27 04:56:47,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:47,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893067790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:47,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893067790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,793 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 13f5a923b0a0364586ae63fe46bf7293/C of 13f5a923b0a0364586ae63fe46bf7293 into 41c55ffbf6154d778a2be422cf4c5a25(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:47,793 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:47,793 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., storeName=13f5a923b0a0364586ae63fe46bf7293/C, priority=13, startTime=1679893007593; duration=0sec 2023-03-27 04:56:47,793 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:47,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:47,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893067799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:47,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893067804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,813 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=742 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/550b713184524422bb39520b165d827d 2023-03-27 04:56:47,855 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:47,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:47,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:47,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:47,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:47,858 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:47,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:47,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:47,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:47,860 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,860 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,861 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:47,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:47,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:47,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:47,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:47,862 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:47,862 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:47,862 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:47,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:47,870 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:47,870 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:47,870 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:47,870 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:47,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:47,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:47,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:47,880 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,880 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,880 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,880 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:47,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31cd5b8c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:47,881 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:47,881 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,883 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,884 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:47,884 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:47,884 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:47,884 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,884 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:47,884 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,884 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:47,884 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,884 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:47,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:47,886 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:47,887 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:47,887 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:47,887 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:47,887 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:47,887 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,887 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,887 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:47,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:47,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:47,888 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:47,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:47,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,891 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:47,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:47,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:47,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:47,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:47,891 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:47,891 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:47,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:47,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:47,991 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:47,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:47,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:47,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:47,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:47,994 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:47,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:47,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:47,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:47,995 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,995 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,996 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,996 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:47,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:47,996 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,996 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:47,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:47,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:47,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:47,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:47,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:47,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:47,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:47,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:47,997 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,997 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:47,997 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:47,997 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:47,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:47,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:47,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:47,998 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:47,998 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,998 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:47,999 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:47,999 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:48,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ea08b6f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:48,000 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:48,000 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,001 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,001 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:48,001 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:48,001 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:48,001 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,001 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:48,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:48,003 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:48,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:48,004 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:48,004 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:48,004 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:48,005 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:48,005 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:48,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:48,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893068002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,006 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,006 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,006 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:48,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,006 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,007 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,010 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,010 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,010 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,010 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,010 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:48,010 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:48,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:48,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:48,010 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:48,010 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:48,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:48,010 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:48,010 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:48,010 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,010 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:48,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:48,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893068009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,023 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/7bda4f0433694a568380e36fe7f03299 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/7bda4f0433694a568380e36fe7f03299 2023-03-27 04:56:48,027 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 13f5a923b0a0364586ae63fe46bf7293/A of 13f5a923b0a0364586ae63fe46bf7293 into 7bda4f0433694a568380e36fe7f03299(size=25.9 K), total size for store is 104.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:48,028 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:48,028 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., storeName=13f5a923b0a0364586ae63fe46bf7293/A, priority=13, startTime=1679893007169; duration=0sec 2023-03-27 04:56:48,028 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:48,110 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:48,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:48,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:48,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:48,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:48,113 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:48,113 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:48,113 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:48,113 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:48,113 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,113 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,114 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:48,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:48,114 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:48,114 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,114 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:48,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:48,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:48,115 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:48,115 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:48,115 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:48,115 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:48,115 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:48,115 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:48,115 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:48,115 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:48,115 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:48,115 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:48,115 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:48,115 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,116 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:48,116 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:48,116 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,116 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,116 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,117 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:48,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5905e44f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:48,118 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:48,118 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,118 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,118 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:48,118 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:48,119 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:48,119 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,119 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:48,120 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:48,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:48,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:48,121 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:48,121 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:48,121 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:48,121 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:48,121 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:48,122 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,122 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,122 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:48,123 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,124 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,126 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,126 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,126 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,126 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:48,126 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:48,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:48,126 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,126 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:48,126 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,126 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:48,126 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:48,126 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:48,126 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:48,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:48,226 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:48,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:48,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:48,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:48,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:48,229 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:48,229 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:48,229 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:48,229 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:48,229 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,229 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,230 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:48,230 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:48,231 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,231 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:48,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:48,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:48,232 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:48,232 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:48,232 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:48,234 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:48,234 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:48,234 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:48,234 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:48,234 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:48,234 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:48,234 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:48,234 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:48,234 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,234 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:48,234 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:48,235 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,235 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,235 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,235 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:48,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33665707[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:48,236 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:48,236 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,237 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,237 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:48,237 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:48,237 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:48,237 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,237 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:48,238 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,238 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:48,238 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:48,238 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:48,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:48,240 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:48,240 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:48,240 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:48,240 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:48,240 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:48,240 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,241 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,241 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:48,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,241 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=742 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/7e3a30119d54461295e8afc366535149 2023-03-27 04:56:48,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,241 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,243 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,246 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/c3a75da504d24408972c73ad6944a503 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/c3a75da504d24408972c73ad6944a503 2023-03-27 04:56:48,246 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,246 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,246 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,246 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,246 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:48,246 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:48,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:48,246 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:48,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:48,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:48,246 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,246 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,246 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:48,246 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:48,246 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:48,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:48,250 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/c3a75da504d24408972c73ad6944a503, entries=200, sequenceid=742, filesize=32.3 K 2023-03-27 04:56:48,250 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/550b713184524422bb39520b165d827d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/550b713184524422bb39520b165d827d 2023-03-27 04:56:48,255 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/550b713184524422bb39520b165d827d, entries=150, sequenceid=742, filesize=11.9 K 2023-03-27 04:56:48,256 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/7e3a30119d54461295e8afc366535149 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/7e3a30119d54461295e8afc366535149 2023-03-27 04:56:48,259 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/7e3a30119d54461295e8afc366535149, entries=150, sequenceid=742, filesize=11.9 K 2023-03-27 04:56:48,260 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 13f5a923b0a0364586ae63fe46bf7293 in 585ms, sequenceid=742, compaction requested=true 2023-03-27 04:56:48,260 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:48,260 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:48,260 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:48,260 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:56:48,260 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:48,260 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-27 04:56:48,261 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:48,261 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:48,261 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. because compaction request was cancelled 2023-03-27 04:56:48,261 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-27 04:56:48,261 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 139836 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:56:48,261 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 13f5a923b0a0364586ae63fe46bf7293/A is initiating minor compaction (all files) 2023-03-27 04:56:48,261 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 13f5a923b0a0364586ae63fe46bf7293/A in TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:48,261 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/7bda4f0433694a568380e36fe7f03299, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/513c298683f944d18794e9ba9a24350f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/2e7651113b0b483db6422021e952177a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/c3a75da504d24408972c73ad6944a503] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp, totalSize=136.6 K 2023-03-27 04:56:48,262 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:48,262 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:48,262 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. because compaction request was cancelled 2023-03-27 04:56:48,262 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 7bda4f0433694a568380e36fe7f03299, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1679893005669 2023-03-27 04:56:48,262 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 513c298683f944d18794e9ba9a24350f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=693, earliestPutTs=1679893006318 2023-03-27 04:56:48,262 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 2e7651113b0b483db6422021e952177a, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=704, earliestPutTs=1679893007448 2023-03-27 04:56:48,263 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting c3a75da504d24408972c73ad6944a503, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=742, earliestPutTs=1679893007560 2023-03-27 04:56:48,279 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 13f5a923b0a0364586ae63fe46bf7293#A#compaction#178 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:48,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:48,308 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:56:48,308 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:48,308 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:48,308 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:48,309 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:48,309 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:48,309 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:48,347 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:48,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:48,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:48,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:48,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:48,351 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:48,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:48,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:48,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:48,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,352 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:48,352 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:48,352 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,353 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:48,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893008347, max timestamp=1679893008354Num uniques 859; 2023-03-27 04:56:48,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:48,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893008347, max timestamp=1679893008354Num uniques 859; 2023-03-27 04:56:48,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:48,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893008347, max timestamp=1679893008354Num uniques 859; 2023-03-27 04:56:48,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:48,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:48,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:48,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:48,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:48,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:48,365 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:48,365 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:48,365 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:48,365 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:48,365 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:48,365 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:48,365 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:48,365 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:48,365 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,365 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:48,365 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:48,367 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,367 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,367 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,367 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:48,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@682c7bdc[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:48,369 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:48,369 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,370 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,370 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:48,370 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:48,370 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:48,370 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,370 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:48,371 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:48,371 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:48,371 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:48,371 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:48,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:48,383 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:48,383 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:48,383 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:48,383 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:48,383 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:48,384 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,384 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,385 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:48,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,385 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,386 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,388 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:48,388 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:48,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:48,388 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:48,389 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,389 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:48,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:48,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:48,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:48,389 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,389 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,389 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,389 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,389 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:48,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:48,389 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:48,389 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:48,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:48,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893068409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,415 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:48,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60406 deadline: 1679893068410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,489 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:48,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:48,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:48,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:48,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:48,492 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:48,492 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:48,492 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:48,492 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:48,492 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,492 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,496 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:48,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:48,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:48,496 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,497 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,497 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:48,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:48,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:48,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:48,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:48,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:48,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:48,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:48,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:48,498 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:48,498 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:48,498 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:48,498 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:48,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:48,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:48,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:48,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:48,499 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,499 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,500 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,500 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:48,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42fd7e45[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:48,501 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:48,501 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,502 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,502 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:48,502 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:48,502 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:48,502 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,502 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:48,503 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:48,503 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,503 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,503 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:48,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:48,505 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:48,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:48,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:48,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:48,505 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:48,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:48,507 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,508 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,510 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:48,510 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:48,510 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:48,510 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:48,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:48,510 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:48,510 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,510 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,510 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,510 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,510 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,510 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:48,510 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:48,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:48,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:48,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60456 deadline: 1679893068516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:48,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60406 deadline: 1679893068516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,610 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:48,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:48,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:48,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:48,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:48,613 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:48,613 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:48,613 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:48,613 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:48,614 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,614 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,620 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:48,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:48,621 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,621 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:48,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:48,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:48,622 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:48,622 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:48,622 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:48,622 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:48,622 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:48,622 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:48,622 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:48,622 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:48,622 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:48,622 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:48,622 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:48,622 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,622 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:48,622 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:48,634 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,634 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,635 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,635 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:48,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@342ccdc9[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:48,636 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:48,636 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,637 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,637 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:48,637 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:48,637 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,637 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:48,637 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:48,638 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:48,638 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:48,638 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:48,638 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:48,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:48,644 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:48,644 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:48,644 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:48,644 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:48,644 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:48,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:48,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,651 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,652 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,654 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:48,654 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:48,654 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:48,654 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:48,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:48,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:48,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:48,654 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,654 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,654 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,654 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,654 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:48,654 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:48,654 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:48,654 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:48,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:48,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60406 deadline: 1679893068720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:48,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60456 deadline: 1679893068721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,725 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303275cc5b1df8f5540978a00b9022057beba to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303275cc5b1df8f5540978a00b9022057beba 2023-03-27 04:56:48,732 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=755, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/1f2b941e1bf04a94bcf9d33a8c37c6db 2023-03-27 04:56:48,747 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=755 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/cd31d2f26bdf49349b915b69296516b3 2023-03-27 04:56:48,754 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:48,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:48,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:48,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:48,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:48,757 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:48,757 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:48,757 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:48,757 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:48,758 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,758 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,758 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:48,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:48,758 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:48,759 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,759 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:48,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:48,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:48,760 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:48,760 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:48,760 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:48,760 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:48,760 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:48,760 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:48,760 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:48,760 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:48,760 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:48,760 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:48,760 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:48,760 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,760 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:48,760 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:48,761 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,761 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,761 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,761 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:48,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35f36a24[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:48,764 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:48,764 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,764 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,764 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:48,764 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:48,764 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:48,764 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,764 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:48,765 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:48,765 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:48,765 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:48,765 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:48,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:48,767 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:48,767 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:48,767 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:48,767 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:48,767 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:48,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,769 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,769 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,771 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,771 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:48,771 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:48,771 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:48,771 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,771 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,771 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:48,771 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:48,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:48,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:48,771 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:48,771 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,772 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:48,772 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:48,872 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:48,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:48,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:48,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:48,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:48,874 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:48,874 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:48,874 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:48,874 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:48,875 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,875 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,876 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,876 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:48,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:48,876 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,876 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:48,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:48,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:48,877 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:48,877 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:48,877 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:48,877 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:48,877 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:48,877 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:48,877 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:48,877 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:48,877 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:48,877 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:48,877 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:48,877 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,877 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:48,877 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:48,878 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,878 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,878 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,878 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:48,879 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:48,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e0a15dd[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:48,879 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,880 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,880 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:48,880 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:48,880 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:48,881 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,881 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:48,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:48,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:48,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:48,882 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:48,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:48,884 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:48,884 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:48,885 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:48,885 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:48,885 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:48,886 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,886 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:48,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,886 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,887 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,888 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,890 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,890 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,890 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,890 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,890 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:48,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:48,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:48,890 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:48,890 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,890 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:48,890 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:48,890 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:48,890 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:48,890 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:48,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:48,990 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:48,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:48,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:48,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:48,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:48,993 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:48,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:48,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:48,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:48,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,995 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,995 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:48,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:48,995 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,995 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:48,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:48,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:48,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:48,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:48,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:48,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:48,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:48,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:48,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:48,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:48,997 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:48,997 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:48,997 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:48,997 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:48,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:48,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:48,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:48,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:48,998 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,998 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:48,998 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:48,998 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:48,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:48,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17ac5b0f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:48,999 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:48,999 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:49,000 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:49,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:49,000 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:49,000 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:49,000 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:49,001 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,001 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:49,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:49,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:49,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:49,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:49,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:49,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:49,007 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:49,007 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:49,007 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:49,007 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:49,007 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:49,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:49,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:49,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:49,018 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:49,019 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,020 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:49,020 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:49,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:49,020 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:49,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:49,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:49,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:49,020 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:49,021 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:49,021 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:49,021 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:49,021 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,021 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:49,021 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:49,021 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:49,021 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:49,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:49,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60406 deadline: 1679893069024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:49,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60456 deadline: 1679893069025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,115 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/55650ad4c8b34e918f14497da93c717f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/55650ad4c8b34e918f14497da93c717f 2023-03-27 04:56:49,120 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 13f5a923b0a0364586ae63fe46bf7293/A of 13f5a923b0a0364586ae63fe46bf7293 into 55650ad4c8b34e918f14497da93c717f(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:49,120 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:49,120 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., storeName=13f5a923b0a0364586ae63fe46bf7293/A, priority=12, startTime=1679893008260; duration=0sec 2023-03-27 04:56:49,120 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:49,121 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:49,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:49,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:49,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:49,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:49,124 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:49,124 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:49,124 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:49,124 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:49,124 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,124 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:49,125 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,125 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:49,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:49,125 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,125 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:49,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:49,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:49,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:49,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:49,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:49,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:49,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:49,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:49,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:49,126 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:49,126 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:49,126 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:49,126 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:49,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:49,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:49,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:49,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:49,127 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:49,128 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:49,128 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:49,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:49,129 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:49,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453c9eb6[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:49,129 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:49,129 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:49,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:49,130 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,130 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:49,130 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:49,130 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:49,130 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:49,130 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,130 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:49,130 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:49,131 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:49,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:49,132 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:49,132 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:49,132 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:49,133 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:49,133 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:49,133 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,133 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:49,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,133 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:49,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:49,134 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:49,135 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,136 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,136 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:49,136 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:49,136 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:49,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:49,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:49,136 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:49,136 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:49,137 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:49,136 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:49,137 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:49,137 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:49,137 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,137 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:49,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:49,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:49,156 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=755 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/a0c0d06c6fea485494ae89ad82ba99fc 2023-03-27 04:56:49,160 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/1f2b941e1bf04a94bcf9d33a8c37c6db as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/1f2b941e1bf04a94bcf9d33a8c37c6db 2023-03-27 04:56:49,164 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/1f2b941e1bf04a94bcf9d33a8c37c6db, entries=150, sequenceid=755, filesize=25.4 K 2023-03-27 04:56:49,165 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/cd31d2f26bdf49349b915b69296516b3 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/cd31d2f26bdf49349b915b69296516b3 2023-03-27 04:56:49,170 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/cd31d2f26bdf49349b915b69296516b3, entries=100, sequenceid=755, filesize=9.5 K 2023-03-27 04:56:49,171 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/a0c0d06c6fea485494ae89ad82ba99fc as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/a0c0d06c6fea485494ae89ad82ba99fc 2023-03-27 04:56:49,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/a0c0d06c6fea485494ae89ad82ba99fc, entries=100, sequenceid=755, filesize=9.5 K 2023-03-27 04:56:49,187 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 13f5a923b0a0364586ae63fe46bf7293 in 879ms, sequenceid=755, compaction requested=true 2023-03-27 04:56:49,187 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:49,187 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:49,187 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-27 04:56:49,187 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:49,187 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:49,188 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:49,188 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:56:49,188 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:56:49,188 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. because compaction request was cancelled 2023-03-27 04:56:49,188 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:56:49,189 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:49,189 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 13f5a923b0a0364586ae63fe46bf7293/B is initiating minor compaction (all files) 2023-03-27 04:56:49,189 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 13f5a923b0a0364586ae63fe46bf7293/B in TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:49,189 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/52e2096c0fa447f2a1362debf6681fab, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/550b713184524422bb39520b165d827d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/cd31d2f26bdf49349b915b69296516b3] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp, totalSize=33.8 K 2023-03-27 04:56:49,189 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 52e2096c0fa447f2a1362debf6681fab, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=704, earliestPutTs=1679893007454 2023-03-27 04:56:49,189 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:56:49,189 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 13f5a923b0a0364586ae63fe46bf7293/C is initiating minor compaction (all files) 2023-03-27 04:56:49,189 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 13f5a923b0a0364586ae63fe46bf7293/C in TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:49,189 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 550b713184524422bb39520b165d827d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=742, earliestPutTs=1679893007560 2023-03-27 04:56:49,190 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/41c55ffbf6154d778a2be422cf4c5a25, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/7e3a30119d54461295e8afc366535149, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/a0c0d06c6fea485494ae89ad82ba99fc] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp, totalSize=33.8 K 2023-03-27 04:56:49,190 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting cd31d2f26bdf49349b915b69296516b3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=755, earliestPutTs=1679893007687 2023-03-27 04:56:49,190 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 41c55ffbf6154d778a2be422cf4c5a25, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=704, earliestPutTs=1679893007454 2023-03-27 04:56:49,191 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 7e3a30119d54461295e8afc366535149, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=742, earliestPutTs=1679893007560 2023-03-27 04:56:49,192 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting a0c0d06c6fea485494ae89ad82ba99fc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=755, earliestPutTs=1679893007687 2023-03-27 04:56:49,203 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 13f5a923b0a0364586ae63fe46bf7293#C#compaction#183 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:49,203 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 13f5a923b0a0364586ae63fe46bf7293#B#compaction#182 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:56:49,237 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:49,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:49,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:49,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:49,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:49,240 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:49,240 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:49,240 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:49,240 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:49,240 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,240 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:49,241 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:49,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:49,241 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:49,241 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,242 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:49,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:49,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:49,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:49,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:49,243 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:49,243 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:49,243 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:49,243 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:49,243 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:49,243 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:49,243 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:49,243 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-27 04:56:49,244 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:49,244 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:49,244 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:49,244 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:49,244 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:49,244 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:49,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:49,529 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:49,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:49,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60406 deadline: 1679893069539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:49,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60456 deadline: 1679893069541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,631 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/755644e4db294c3da7774f27054d0a9b as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/755644e4db294c3da7774f27054d0a9b 2023-03-27 04:56:49,639 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 13f5a923b0a0364586ae63fe46bf7293/C of 13f5a923b0a0364586ae63fe46bf7293 into 755644e4db294c3da7774f27054d0a9b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:49,639 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:49,639 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., storeName=13f5a923b0a0364586ae63fe46bf7293/C, priority=13, startTime=1679893009188; duration=0sec 2023-03-27 04:56:49,639 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:49,642 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/c326acc5e5d74678ac59f718f01a1f38 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/c326acc5e5d74678ac59f718f01a1f38 2023-03-27 04:56:49,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:49,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60406 deadline: 1679893069642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:49,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60456 deadline: 1679893069646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,650 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 13f5a923b0a0364586ae63fe46bf7293/B of 13f5a923b0a0364586ae63fe46bf7293 into c326acc5e5d74678ac59f718f01a1f38(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:56:49,650 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:49,650 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., storeName=13f5a923b0a0364586ae63fe46bf7293/B, priority=13, startTime=1679893009187; duration=0sec 2023-03-27 04:56:49,650 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:49,664 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032741a1b36e98224ecea9bc881242dc51c8 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032741a1b36e98224ecea9bc881242dc51c8 2023-03-27 04:56:49,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:49,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893069808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:49,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893069809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,813 DEBUG [Thread-985] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., hostname=jenkins-hbase20.apache.org,44923,1679892955505, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:56:49,815 DEBUG [Thread-981] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293., hostname=jenkins-hbase20.apache.org,44923,1679892955505, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:56:49,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:49,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60406 deadline: 1679893069845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:49,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:49,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60456 deadline: 1679893069850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,069 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=792, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/cc0fc91b3b4146559b98065c9ce90080 2023-03-27 04:56:50,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:50,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60406 deadline: 1679893070149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:50,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60456 deadline: 1679893070156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,480 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=792 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/807804a2cf4a4eae9df1a9f235056ef9 2023-03-27 04:56:50,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:50,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60406 deadline: 1679893070653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:50,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60456 deadline: 1679893070659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,889 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=792 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/988d25dd894f4e59b644ca5526412fe4 2023-03-27 04:56:50,892 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/cc0fc91b3b4146559b98065c9ce90080 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/cc0fc91b3b4146559b98065c9ce90080 2023-03-27 04:56:50,899 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/cc0fc91b3b4146559b98065c9ce90080, entries=150, sequenceid=792, filesize=25.4 K 2023-03-27 04:56:50,900 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/807804a2cf4a4eae9df1a9f235056ef9 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/807804a2cf4a4eae9df1a9f235056ef9 2023-03-27 04:56:50,905 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/807804a2cf4a4eae9df1a9f235056ef9, entries=150, sequenceid=792, filesize=11.9 K 2023-03-27 04:56:50,905 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/988d25dd894f4e59b644ca5526412fe4 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/988d25dd894f4e59b644ca5526412fe4 2023-03-27 04:56:50,909 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/988d25dd894f4e59b644ca5526412fe4, entries=150, sequenceid=792, filesize=11.9 K 2023-03-27 04:56:50,910 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 13f5a923b0a0364586ae63fe46bf7293 in 1666ms, sequenceid=792, compaction requested=true 2023-03-27 04:56:50,910 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:50,910 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:50,910 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:50,910 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:50,910 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,910 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:50,910 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:50,911 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:50,911 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:50,911 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:50,911 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:50,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:50,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:50,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bb06e7f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:50,912 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:50,912 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:50,913 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:50,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:50,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:50,913 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:50,913 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:50,913 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:50,913 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,913 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:50,913 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,913 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:50,913 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:50,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:50,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:50,913 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:50,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:50,915 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:50,915 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:50,915 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:50,915 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:50,915 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:50,916 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:50,916 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:50,917 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:50,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:50,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:50,917 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:50,918 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,920 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,920 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:50,920 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:50,920 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:50,920 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:50,920 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:50,920 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:50,920 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:50,920 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:50,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:50,920 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:50,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:50,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:50,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:50,921 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:50,921 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:50,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:51,021 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:51,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:51,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:51,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:51,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:51,024 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:51,024 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:51,024 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:51,024 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:51,024 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:51,024 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:51,025 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:51,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:51,025 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:51,025 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,025 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:51,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:51,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:51,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:51,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:51,026 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:51,026 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:51,026 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:51,026 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:51,026 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:51,026 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:51,026 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:51,026 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:56:51,026 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:51,026 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:51,026 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:51,027 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:51,027 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:51,027 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:51,438 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327176185de8a5746a882298b6c3ee2395f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327176185de8a5746a882298b6c3ee2395f 2023-03-27 04:56:51,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:51,664 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:51,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893011681, max timestamp=1679893011688Num uniques 859; 2023-03-27 04:56:51,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893011681, max timestamp=1679893011688Num uniques 859; 2023-03-27 04:56:51,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:51,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893011681, max timestamp=1679893011688Num uniques 859; 2023-03-27 04:56:51,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:51,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:56:51,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:51,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:51,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60456 deadline: 1679893071735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60406 deadline: 1679893071735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:51,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60406 deadline: 1679893071840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:51,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60456 deadline: 1679893071840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,844 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=804, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/173274d349f84616a512d1e8fe8b47e3 2023-03-27 04:56:51,853 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=804 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/8812208700154093a3cf1f84533def2a 2023-03-27 04:56:51,863 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=804 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/955c3ded156441b6817711ff2052b6ee 2023-03-27 04:56:51,873 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/173274d349f84616a512d1e8fe8b47e3 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/173274d349f84616a512d1e8fe8b47e3 2023-03-27 04:56:51,877 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/173274d349f84616a512d1e8fe8b47e3, entries=150, sequenceid=804, filesize=25.4 K 2023-03-27 04:56:51,878 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/8812208700154093a3cf1f84533def2a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/8812208700154093a3cf1f84533def2a 2023-03-27 04:56:51,881 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/8812208700154093a3cf1f84533def2a, entries=150, sequenceid=804, filesize=11.9 K 2023-03-27 04:56:51,881 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/955c3ded156441b6817711ff2052b6ee as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/955c3ded156441b6817711ff2052b6ee 2023-03-27 04:56:51,885 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/955c3ded156441b6817711ff2052b6ee, entries=150, sequenceid=804, filesize=11.9 K 2023-03-27 04:56:51,885 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 13f5a923b0a0364586ae63fe46bf7293 in 859ms, sequenceid=804, compaction requested=true 2023-03-27 04:56:51,885 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:51,885 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:51,886 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:51,886 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:51,886 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,886 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:51,886 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:51,886 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:51,887 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:51,887 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:51,887 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:51,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:51,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fa2a29[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:51,888 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:51,888 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:51,889 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:51,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:51,889 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:51,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:51,889 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:51,889 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:51,889 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:51,889 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:51,889 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:51,889 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:51,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:51,891 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:51,891 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:51,891 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:51,891 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:51,891 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:51,893 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:51,893 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:51,893 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:51,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:51,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:51,893 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,894 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,895 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,895 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:51,895 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:51,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:51,895 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:51,895 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:51,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:51,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:51,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:51,895 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:51,896 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:51,896 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:51,896 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:51,896 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:51,896 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:51,896 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:51,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:51,996 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:51,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:51,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:51,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:51,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:51,999 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:51,999 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:51,999 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:51,999 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:52,000 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:52,000 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:52,000 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:52,000 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:52,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:52,000 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:52,001 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:52,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:52,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:52,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:52,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:52,001 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:52,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:52,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:52,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:52,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:52,002 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:52,002 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:52,002 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-27 04:56:52,002 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:52,002 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:52,002 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:52,002 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:52,002 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:52,002 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:52,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:52,049 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:52,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:52,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60456 deadline: 1679893072056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:52,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:52,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60406 deadline: 1679893072058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:52,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:52,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60456 deadline: 1679893072159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:52,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:52,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60406 deadline: 1679893072162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:52,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:52,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60406 deadline: 1679893072363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:52,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:52,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60456 deadline: 1679893072363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:52,415 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327e6f961be87d3467296ec9f69e6d90be4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e6f961be87d3467296ec9f69e6d90be4 2023-03-27 04:56:52,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:52,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60406 deadline: 1679893072665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:52,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:52,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60456 deadline: 1679893072668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:52,820 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=840, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/a6100654eda14442beb5ce2d21546cba 2023-03-27 04:56:53,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:53,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60406 deadline: 1679893073167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:53,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60456 deadline: 1679893073176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,231 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=840 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/79df079dc70a482f9b155d518d3e9909 2023-03-27 04:56:53,251 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=840 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/162546b12efb4617a463709c23d16965 2023-03-27 04:56:53,256 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/a6100654eda14442beb5ce2d21546cba as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/a6100654eda14442beb5ce2d21546cba 2023-03-27 04:56:53,261 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/a6100654eda14442beb5ce2d21546cba, entries=150, sequenceid=840, filesize=25.4 K 2023-03-27 04:56:53,262 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/79df079dc70a482f9b155d518d3e9909 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/79df079dc70a482f9b155d518d3e9909 2023-03-27 04:56:53,266 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/79df079dc70a482f9b155d518d3e9909, entries=150, sequenceid=840, filesize=11.9 K 2023-03-27 04:56:53,266 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/162546b12efb4617a463709c23d16965 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/162546b12efb4617a463709c23d16965 2023-03-27 04:56:53,270 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/162546b12efb4617a463709c23d16965, entries=150, sequenceid=840, filesize=11.9 K 2023-03-27 04:56:53,271 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 13f5a923b0a0364586ae63fe46bf7293 in 1269ms, sequenceid=840, compaction requested=true 2023-03-27 04:56:53,272 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:53,272 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:53,272 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:53,272 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:53,272 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,272 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:53,272 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:53,273 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:53,273 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:53,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:53,273 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:53,273 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:53,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:53,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:53,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:53,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:53,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70fb7dba[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:53,274 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:53,274 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:53,275 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:53,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:53,275 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:53,275 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:53,275 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:53,275 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,275 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:53,275 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:53,276 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:53,276 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:53,276 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:53,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:53,279 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:53,279 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:53,279 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:53,279 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:53,279 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:53,287 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:53,287 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:53,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:53,287 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:53,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:53,287 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:53,289 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,307 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:53,307 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:53,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:53,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:53,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:53,307 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:53,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:53,307 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:53,307 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:53,307 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:53,307 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:53,307 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:53,307 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:53,307 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:53,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:53,307 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:53,307 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:53,407 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:53,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:53,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:53,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:53,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:53,410 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:53,411 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:53,411 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:53,411 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:53,411 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:53,411 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:53,411 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:53,411 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:53,412 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,412 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:53,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:53,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:53,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:53,413 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:53,413 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:53,413 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:53,413 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:53,413 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:53,413 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:53,413 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:53,413 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:56:53,413 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:53,413 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:53,413 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:53,413 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:53,413 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:53,414 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:53,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:53,826 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:53,831 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032797205f1b46d643418456e5e9e9b2ce64 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032797205f1b46d643418456e5e9e9b2ce64 2023-03-27 04:56:53,836 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=850, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/a65a03bd380540d68b225c699afcb780 2023-03-27 04:56:53,845 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=850 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/105f40e521134403ad0218b03b77ecf3 2023-03-27 04:56:53,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893013852, max timestamp=1679893013861Num uniques 859; 2023-03-27 04:56:53,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893013852, max timestamp=1679893013861Num uniques 859; 2023-03-27 04:56:53,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893013852, max timestamp=1679893013861Num uniques 859; 2023-03-27 04:56:53,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:53,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:53,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:56:53,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:53,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893073905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:53,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:53,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893073905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:54,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893074008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:54,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893074008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:54,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60406 deadline: 1679893074170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:54,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60456 deadline: 1679893074179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:54,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893074210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:54,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893074210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,255 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=850 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/1f22a923ad5445b0ba5e1e0e4d1340c0 2023-03-27 04:56:54,259 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/a65a03bd380540d68b225c699afcb780 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/a65a03bd380540d68b225c699afcb780 2023-03-27 04:56:54,262 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/a65a03bd380540d68b225c699afcb780, entries=150, sequenceid=850, filesize=25.4 K 2023-03-27 04:56:54,263 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/105f40e521134403ad0218b03b77ecf3 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/105f40e521134403ad0218b03b77ecf3 2023-03-27 04:56:54,267 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/105f40e521134403ad0218b03b77ecf3, entries=150, sequenceid=850, filesize=11.9 K 2023-03-27 04:56:54,272 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/1f22a923ad5445b0ba5e1e0e4d1340c0 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/1f22a923ad5445b0ba5e1e0e4d1340c0 2023-03-27 04:56:54,278 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/1f22a923ad5445b0ba5e1e0e4d1340c0, entries=150, sequenceid=850, filesize=11.9 K 2023-03-27 04:56:54,279 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 13f5a923b0a0364586ae63fe46bf7293 in 866ms, sequenceid=850, compaction requested=true 2023-03-27 04:56:54,280 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:54,280 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:54,280 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:54,280 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:54,280 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,280 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:54,280 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:54,281 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:54,282 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:54,283 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:54,283 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:54,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:54,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5113943d[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:54,284 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:54,284 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:54,285 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:54,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:54,285 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:54,285 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:54,285 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:54,285 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,285 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:54,286 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:54,286 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,286 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:54,286 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:54,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:54,289 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:54,289 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:54,289 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:54,289 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:54,289 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:54,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:54,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:54,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:54,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:54,290 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,292 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,293 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:54,293 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:54,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:54,293 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:54,294 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:54,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:54,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:54,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:54,294 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:54,294 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:54,294 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:54,294 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:54,294 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:54,294 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:54,294 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:54,294 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:54,394 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:54,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:54,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:56:54,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:56:54,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:56:54,397 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:56:54,397 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:54,397 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:56:54,397 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:56:54,398 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:54,398 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:54,398 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,398 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:54,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:54,399 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,399 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:56:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:54,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:56:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:54,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:56:54,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:56:54,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:56:54,400 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:56:54,400 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:56:54,400 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:56:54,400 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:56:54,400 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:56:54,400 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:54,400 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. started... 2023-03-27 04:56:54,400 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-27 04:56:54,400 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:54,400 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:54,400 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:54,400 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:54,400 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:54,401 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:56:54,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:54,517 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. as already flushing 2023-03-27 04:56:54,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:54,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893074530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:54,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893074530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:54,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893074637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:54,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893074637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,812 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303271293a4742b9a4e92963ec341ec0513d6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303271293a4742b9a4e92963ec341ec0513d6 2023-03-27 04:56:54,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:54,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893074839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:54,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:54,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893074840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:55,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:55,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893075141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:55,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:55,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893075145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:55,180 DEBUG [Thread-996] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52902392 to 127.0.0.1:61059 2023-03-27 04:56:55,180 DEBUG [Thread-996] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:55,180 DEBUG [Thread-992] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50feca84 to 127.0.0.1:61059 2023-03-27 04:56:55,180 DEBUG [Thread-992] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:55,180 DEBUG [Thread-994] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bd79dff to 127.0.0.1:61059 2023-03-27 04:56:55,180 DEBUG [Thread-994] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:55,182 DEBUG [Thread-990] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x486fba63 to 127.0.0.1:61059 2023-03-27 04:56:55,182 DEBUG [Thread-990] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:55,218 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=887, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/3f2e27212be94050a2e29cc3566461c9 2023-03-27 04:56:55,634 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=887 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/bf3805566c514c6f8071315e5889737a 2023-03-27 04:56:55,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:55,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60440 deadline: 1679893075647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:55,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:56:55,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60418 deadline: 1679893075650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:56,053 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=887 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/d8dc65406ccf4d2dbaa31023866b7ff8 2023-03-27 04:56:56,064 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/3f2e27212be94050a2e29cc3566461c9 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/3f2e27212be94050a2e29cc3566461c9 2023-03-27 04:56:56,069 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/3f2e27212be94050a2e29cc3566461c9, entries=150, sequenceid=887, filesize=25.4 K 2023-03-27 04:56:56,071 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/bf3805566c514c6f8071315e5889737a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/bf3805566c514c6f8071315e5889737a 2023-03-27 04:56:56,077 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/bf3805566c514c6f8071315e5889737a, entries=150, sequenceid=887, filesize=11.9 K 2023-03-27 04:56:56,077 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/d8dc65406ccf4d2dbaa31023866b7ff8 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/d8dc65406ccf4d2dbaa31023866b7ff8 2023-03-27 04:56:56,083 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/d8dc65406ccf4d2dbaa31023866b7ff8, entries=150, sequenceid=887, filesize=11.9 K 2023-03-27 04:56:56,083 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for 13f5a923b0a0364586ae63fe46bf7293 in 1683ms, sequenceid=887, compaction requested=true 2023-03-27 04:56:56,083 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:56,084 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:56,084 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:56:56,084 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:56:56,084 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:56,084 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:56:56,084 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:56:56,085 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:56,085 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:56,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:56,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:56:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:56,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:56,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:56,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:56:56,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39b8cbf6[Count = 0] remaining members to acquire global barrier 2023-03-27 04:56:56,088 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:56:56,088 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:56,089 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:56,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:56,089 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:56:56,089 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:56:56,089 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:56:56,089 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:56,089 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:56:56,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:56:56,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:56,090 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:56,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:56:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:56:56,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:56:56,093 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:56:56,094 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:56:56,094 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:56:56,094 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:56:56,094 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:56:56,095 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:56,095 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:56:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:56:56,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:56,095 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:56,096 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:56,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:56:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:56:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:56,097 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:56:56,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:56:56,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:56,100 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:56,100 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:56,100 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:56:56,100 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:56,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:56:56,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:56:56,100 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:56:56,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:56:56,100 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:56,100 DEBUG [Thread-989] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:56:56,100 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:56:56,101 DEBUG [Thread-989] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:56:56,100 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:56:56,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:56:56,100 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:56:56,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:56:56,101 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:56:56,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:56:56,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:56:56,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:56,184 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-27 04:56:56,184 DEBUG [Thread-979] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e3bbbd0 to 127.0.0.1:61059 2023-03-27 04:56:56,184 DEBUG [Thread-979] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:56,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:56,186 DEBUG [Thread-987] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7be739cc to 127.0.0.1:61059 2023-03-27 04:56:56,186 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:56,186 DEBUG [Thread-987] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:56,186 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:56,187 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:56,187 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:56,187 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:56,201 DEBUG [Thread-989] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:56:56,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:56:56,351 DEBUG [Thread-983] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b3311fc to 127.0.0.1:61059 2023-03-27 04:56:56,351 DEBUG [Thread-983] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:56,606 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303278619a3084e704d718dbaa492f3bc1e47 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303278619a3084e704d718dbaa492f3bc1e47 2023-03-27 04:56:56,661 DEBUG [Thread-985] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7acc1387 to 127.0.0.1:61059 2023-03-27 04:56:56,661 DEBUG [Thread-981] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d781def to 127.0.0.1:61059 2023-03-27 04:56:56,661 DEBUG [Thread-981] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:56,661 DEBUG [Thread-985] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:56,661 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-27 04:56:56,662 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 100 2023-03-27 04:56:56,662 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 284 2023-03-27 04:56:56,662 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 29 2023-03-27 04:56:56,664 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 304 2023-03-27 04:56:56,664 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 104 2023-03-27 04:56:56,665 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-27 04:56:56,665 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(398): read 5264 2023-03-27 04:56:56,665 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(398): read 5425 2023-03-27 04:56:56,665 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-27 04:56:56,665 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(402): scanned 2321 2023-03-27 04:56:56,665 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(403): verified 6963 rows 2023-03-27 04:56:56,665 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(402): scanned 2304 2023-03-27 04:56:56,665 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(403): verified 6912 rows 2023-03-27 04:56:56,665 INFO [Listener at localhost.localdomain/43311] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-27 04:56:56,665 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7306f12a to 127.0.0.1:61059 2023-03-27 04:56:56,666 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:56:56,667 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-27 04:56:56,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-27 04:56:56,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-27 04:56:56,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-27 04:56:56,673 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893016673"}]},"ts":"1679893016673"} 2023-03-27 04:56:56,674 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-27 04:56:56,678 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-27 04:56:56,679 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=13f5a923b0a0364586ae63fe46bf7293, UNASSIGN}] 2023-03-27 04:56:56,680 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=13f5a923b0a0364586ae63fe46bf7293, UNASSIGN 2023-03-27 04:56:56,680 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=13f5a923b0a0364586ae63fe46bf7293, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:56:56,680 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893016680"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679893016680"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679893016680"}]},"ts":"1679893016680"} 2023-03-27 04:56:56,681 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:56:56,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-27 04:56:56,833 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:56,834 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 13f5a923b0a0364586ae63fe46bf7293, disabling compactions & flushes 2023-03-27 04:56:56,834 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:56,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-27 04:56:57,013 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=900, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/d70b2ae64f04472a95a9794750a58cc9 2023-03-27 04:56:57,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-27 04:56:57,428 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=900 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/0bcff1d73a7a4a1db3d09241a31767d2 2023-03-27 04:56:57,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-27 04:56:57,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=900 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/fc448cfd1ef84937baeed8e8fa745ffb 2023-03-27 04:56:57,858 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/d70b2ae64f04472a95a9794750a58cc9 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/d70b2ae64f04472a95a9794750a58cc9 2023-03-27 04:56:57,864 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/d70b2ae64f04472a95a9794750a58cc9, entries=150, sequenceid=900, filesize=25.4 K 2023-03-27 04:56:57,865 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/0bcff1d73a7a4a1db3d09241a31767d2 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/0bcff1d73a7a4a1db3d09241a31767d2 2023-03-27 04:56:57,870 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/0bcff1d73a7a4a1db3d09241a31767d2, entries=150, sequenceid=900, filesize=11.9 K 2023-03-27 04:56:57,871 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/fc448cfd1ef84937baeed8e8fa745ffb as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/fc448cfd1ef84937baeed8e8fa745ffb 2023-03-27 04:56:57,876 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/fc448cfd1ef84937baeed8e8fa745ffb, entries=150, sequenceid=900, filesize=11.9 K 2023-03-27 04:56:57,877 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=20.13 KB/20610 for 13f5a923b0a0364586ae63fe46bf7293 in 1693ms, sequenceid=900, compaction requested=true 2023-03-27 04:56:57,877 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:57,877 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:57,877 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:57,877 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. because compaction request was cancelled 2023-03-27 04:56:57,877 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. after waiting 0 ms 2023-03-27 04:56:57,877 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:57,878 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:57,878 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:56:57,878 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 13f5a923b0a0364586ae63fe46bf7293 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-27 04:56:57,878 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. because compaction request was cancelled 2023-03-27 04:56:57,878 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. because compaction request was cancelled 2023-03-27 04:56:57,878 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=A 2023-03-27 04:56:57,878 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:56:57,878 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:57,878 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=B 2023-03-27 04:56:57,878 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:57,878 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 13f5a923b0a0364586ae63fe46bf7293, store=C 2023-03-27 04:56:57,879 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:56:58,298 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327e164065fd4a4403b880690842b89c039 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e164065fd4a4403b880690842b89c039 2023-03-27 04:56:58,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=906, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/8e010ebba070429bbd51064089fce9ce 2023-03-27 04:56:58,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-27 04:56:59,127 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=906 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/ab47076d95044ea48ac1f94f614475a1 2023-03-27 04:56:59,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=906 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/d7bcd1b394734be3b4b42b430b792088 2023-03-27 04:56:59,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/A/8e010ebba070429bbd51064089fce9ce as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/8e010ebba070429bbd51064089fce9ce 2023-03-27 04:56:59,158 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/8e010ebba070429bbd51064089fce9ce, entries=50, sequenceid=906, filesize=11.7 K 2023-03-27 04:56:59,159 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/B/ab47076d95044ea48ac1f94f614475a1 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/ab47076d95044ea48ac1f94f614475a1 2023-03-27 04:56:59,164 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/ab47076d95044ea48ac1f94f614475a1, entries=50, sequenceid=906, filesize=7.1 K 2023-03-27 04:56:59,164 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/.tmp/C/d7bcd1b394734be3b4b42b430b792088 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/d7bcd1b394734be3b4b42b430b792088 2023-03-27 04:56:59,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/d7bcd1b394734be3b4b42b430b792088, entries=50, sequenceid=906, filesize=7.1 K 2023-03-27 04:56:59,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 13f5a923b0a0364586ae63fe46bf7293 in 1293ms, sequenceid=906, compaction requested=true 2023-03-27 04:56:59,185 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0a6dec5e88144493baec26c739d2597c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/c7ba3c92e24940aba1c6c7293dec42f8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/675db2aac9254823825689568e21cb4b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/349b2bac84194ef78f3a25267b18d53b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/e4c2f0686a3a48639f989c12cf79e855, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0d4b69010b164bf28dd841228b8c40c1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/58f6022cbab9452681e688762d66633f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/bdbbd8ea598449a59b2ba28a75f9a190, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/18431024500d491ca45e367f7400d7b3, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/e42f68754f2a4746ab368d5256c07439, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/af08e79bcb6541299482ac29d930b407, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0d15f62738234de8b42053b759074801, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/4273d343f5ae46f9967cbba4acc0d455, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/7bda4f0433694a568380e36fe7f03299, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/513c298683f944d18794e9ba9a24350f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/2e7651113b0b483db6422021e952177a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/c3a75da504d24408972c73ad6944a503] to archive 2023-03-27 04:56:59,186 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:56:59,188 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0a6dec5e88144493baec26c739d2597c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0a6dec5e88144493baec26c739d2597c 2023-03-27 04:56:59,189 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/c7ba3c92e24940aba1c6c7293dec42f8 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/c7ba3c92e24940aba1c6c7293dec42f8 2023-03-27 04:56:59,191 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/675db2aac9254823825689568e21cb4b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/675db2aac9254823825689568e21cb4b 2023-03-27 04:56:59,192 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/349b2bac84194ef78f3a25267b18d53b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/349b2bac84194ef78f3a25267b18d53b 2023-03-27 04:56:59,194 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/e4c2f0686a3a48639f989c12cf79e855 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/e4c2f0686a3a48639f989c12cf79e855 2023-03-27 04:56:59,195 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0d4b69010b164bf28dd841228b8c40c1 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0d4b69010b164bf28dd841228b8c40c1 2023-03-27 04:56:59,196 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/58f6022cbab9452681e688762d66633f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/58f6022cbab9452681e688762d66633f 2023-03-27 04:56:59,197 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/bdbbd8ea598449a59b2ba28a75f9a190 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/bdbbd8ea598449a59b2ba28a75f9a190 2023-03-27 04:56:59,199 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/18431024500d491ca45e367f7400d7b3 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/18431024500d491ca45e367f7400d7b3 2023-03-27 04:56:59,200 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/e42f68754f2a4746ab368d5256c07439 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/e42f68754f2a4746ab368d5256c07439 2023-03-27 04:56:59,201 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/af08e79bcb6541299482ac29d930b407 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/af08e79bcb6541299482ac29d930b407 2023-03-27 04:56:59,202 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0d15f62738234de8b42053b759074801 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/0d15f62738234de8b42053b759074801 2023-03-27 04:56:59,203 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/4273d343f5ae46f9967cbba4acc0d455 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/4273d343f5ae46f9967cbba4acc0d455 2023-03-27 04:56:59,204 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/7bda4f0433694a568380e36fe7f03299 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/7bda4f0433694a568380e36fe7f03299 2023-03-27 04:56:59,205 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/513c298683f944d18794e9ba9a24350f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/513c298683f944d18794e9ba9a24350f 2023-03-27 04:56:59,206 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/2e7651113b0b483db6422021e952177a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/2e7651113b0b483db6422021e952177a 2023-03-27 04:56:59,207 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/c3a75da504d24408972c73ad6944a503 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/c3a75da504d24408972c73ad6944a503 2023-03-27 04:56:59,215 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/6bbd2cb2203b4fd39c39cfd31ff40ab7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/ad978f4bec9e456ba092c2655bc0d5b3, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/b7e5b545c2fa4d719c64312b76fd05ce, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/87c0a7193236450db9884f33e6f85d9b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/941011a49891441e9db5de488c755db7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/29cbccac02854bb0985a808e65c5fbc0, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/d15a2f5100d749a697a5c53308057a83, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/b54216b70e644931bb65a88a17c82d43, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/09b99d761bfa408691741d4ea358ca32, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/6f60b67e068d414d973c522391cbce9f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/3b4517319f034ae39ed142cef9c68a91, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/ee7e717125fa448ab5caf0daa8c88cfb, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/8d2c9c00df1f4d1ca61b9e1a40958712, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/8953d36b53bf4aff98a98808ee194104, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/2d57901949814c59b1b283ddd5416049, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/72ca5eb9bd15444da6c83cee3c3f5af4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/52e2096c0fa447f2a1362debf6681fab, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/550b713184524422bb39520b165d827d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/cd31d2f26bdf49349b915b69296516b3] to archive 2023-03-27 04:56:59,215 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:56:59,217 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/6bbd2cb2203b4fd39c39cfd31ff40ab7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/6bbd2cb2203b4fd39c39cfd31ff40ab7 2023-03-27 04:56:59,217 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/ad978f4bec9e456ba092c2655bc0d5b3 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/ad978f4bec9e456ba092c2655bc0d5b3 2023-03-27 04:56:59,218 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/b7e5b545c2fa4d719c64312b76fd05ce to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/b7e5b545c2fa4d719c64312b76fd05ce 2023-03-27 04:56:59,219 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/87c0a7193236450db9884f33e6f85d9b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/87c0a7193236450db9884f33e6f85d9b 2023-03-27 04:56:59,220 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/941011a49891441e9db5de488c755db7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/941011a49891441e9db5de488c755db7 2023-03-27 04:56:59,221 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/29cbccac02854bb0985a808e65c5fbc0 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/29cbccac02854bb0985a808e65c5fbc0 2023-03-27 04:56:59,222 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/d15a2f5100d749a697a5c53308057a83 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/d15a2f5100d749a697a5c53308057a83 2023-03-27 04:56:59,223 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/b54216b70e644931bb65a88a17c82d43 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/b54216b70e644931bb65a88a17c82d43 2023-03-27 04:56:59,224 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/09b99d761bfa408691741d4ea358ca32 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/09b99d761bfa408691741d4ea358ca32 2023-03-27 04:56:59,225 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/6f60b67e068d414d973c522391cbce9f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/6f60b67e068d414d973c522391cbce9f 2023-03-27 04:56:59,226 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/3b4517319f034ae39ed142cef9c68a91 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/3b4517319f034ae39ed142cef9c68a91 2023-03-27 04:56:59,227 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/ee7e717125fa448ab5caf0daa8c88cfb to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/ee7e717125fa448ab5caf0daa8c88cfb 2023-03-27 04:56:59,228 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/8d2c9c00df1f4d1ca61b9e1a40958712 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/8d2c9c00df1f4d1ca61b9e1a40958712 2023-03-27 04:56:59,228 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/8953d36b53bf4aff98a98808ee194104 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/8953d36b53bf4aff98a98808ee194104 2023-03-27 04:56:59,229 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/2d57901949814c59b1b283ddd5416049 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/2d57901949814c59b1b283ddd5416049 2023-03-27 04:56:59,230 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/72ca5eb9bd15444da6c83cee3c3f5af4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/72ca5eb9bd15444da6c83cee3c3f5af4 2023-03-27 04:56:59,231 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/52e2096c0fa447f2a1362debf6681fab to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/52e2096c0fa447f2a1362debf6681fab 2023-03-27 04:56:59,231 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/550b713184524422bb39520b165d827d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/550b713184524422bb39520b165d827d 2023-03-27 04:56:59,232 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/cd31d2f26bdf49349b915b69296516b3 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/cd31d2f26bdf49349b915b69296516b3 2023-03-27 04:56:59,235 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/147f964d333b4c2186fc691938f6417b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/762988dfac854405b7d7ed918ac7abee, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e1f60d59b4e2409686e6c8ecb8be45ff, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/cb7f36d0a7a04284b6dc25e9ed62f221, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/56c33b2caf8c403ebbf240782a62f8b7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/a71d644a1c0b4bef976507b42b581b83, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/fdacab010600457bbd664f6ba79ed5e9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/ae6e633cca4e4d059a6a40ace66050a5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e7300ba3e1ef48a38559ed64bf55efa7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/6e7231408c804f23bad9baeaab471bad, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/87e6348612304f31b771404f66793685, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/3a198304f3244b79af6fb4ad42312bda, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/0ae15e9c1b5747a3a40261f274c31952, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e7662f89656944d091a2678b93dde3a3, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/f84d56c32b004163a1fb5d4bbc9e9c35, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/41c55ffbf6154d778a2be422cf4c5a25, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/0178dc476ff74650ac96910ed0b92590, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/7e3a30119d54461295e8afc366535149, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/a0c0d06c6fea485494ae89ad82ba99fc] to archive 2023-03-27 04:56:59,235 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:56:59,236 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/147f964d333b4c2186fc691938f6417b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/147f964d333b4c2186fc691938f6417b 2023-03-27 04:56:59,237 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/762988dfac854405b7d7ed918ac7abee to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/762988dfac854405b7d7ed918ac7abee 2023-03-27 04:56:59,238 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e1f60d59b4e2409686e6c8ecb8be45ff to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e1f60d59b4e2409686e6c8ecb8be45ff 2023-03-27 04:56:59,239 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/cb7f36d0a7a04284b6dc25e9ed62f221 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/cb7f36d0a7a04284b6dc25e9ed62f221 2023-03-27 04:56:59,240 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/56c33b2caf8c403ebbf240782a62f8b7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/56c33b2caf8c403ebbf240782a62f8b7 2023-03-27 04:56:59,241 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/a71d644a1c0b4bef976507b42b581b83 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/a71d644a1c0b4bef976507b42b581b83 2023-03-27 04:56:59,242 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/fdacab010600457bbd664f6ba79ed5e9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/fdacab010600457bbd664f6ba79ed5e9 2023-03-27 04:56:59,242 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/ae6e633cca4e4d059a6a40ace66050a5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/ae6e633cca4e4d059a6a40ace66050a5 2023-03-27 04:56:59,243 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e7300ba3e1ef48a38559ed64bf55efa7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e7300ba3e1ef48a38559ed64bf55efa7 2023-03-27 04:56:59,244 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/6e7231408c804f23bad9baeaab471bad to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/6e7231408c804f23bad9baeaab471bad 2023-03-27 04:56:59,245 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/87e6348612304f31b771404f66793685 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/87e6348612304f31b771404f66793685 2023-03-27 04:56:59,246 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/3a198304f3244b79af6fb4ad42312bda to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/3a198304f3244b79af6fb4ad42312bda 2023-03-27 04:56:59,247 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/0ae15e9c1b5747a3a40261f274c31952 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/0ae15e9c1b5747a3a40261f274c31952 2023-03-27 04:56:59,248 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e7662f89656944d091a2678b93dde3a3 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/e7662f89656944d091a2678b93dde3a3 2023-03-27 04:56:59,249 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/f84d56c32b004163a1fb5d4bbc9e9c35 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/f84d56c32b004163a1fb5d4bbc9e9c35 2023-03-27 04:56:59,250 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/41c55ffbf6154d778a2be422cf4c5a25 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/41c55ffbf6154d778a2be422cf4c5a25 2023-03-27 04:56:59,251 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/0178dc476ff74650ac96910ed0b92590 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/0178dc476ff74650ac96910ed0b92590 2023-03-27 04:56:59,252 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/7e3a30119d54461295e8afc366535149 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/7e3a30119d54461295e8afc366535149 2023-03-27 04:56:59,253 DEBUG [StoreCloser-TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/a0c0d06c6fea485494ae89ad82ba99fc to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/a0c0d06c6fea485494ae89ad82ba99fc 2023-03-27 04:56:59,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/recovered.edits/909.seqid, newMaxSeqId=909, maxSeqId=4 2023-03-27 04:56:59,261 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293. 2023-03-27 04:56:59,261 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 13f5a923b0a0364586ae63fe46bf7293: 2023-03-27 04:56:59,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:56:59,264 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=13f5a923b0a0364586ae63fe46bf7293, regionState=CLOSED 2023-03-27 04:56:59,264 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893019263"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679893019263"}]},"ts":"1679893019263"} 2023-03-27 04:56:59,266 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-27 04:56:59,266 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 13f5a923b0a0364586ae63fe46bf7293, server=jenkins-hbase20.apache.org,44923,1679892955505 in 2.5840 sec 2023-03-27 04:56:59,267 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-27 04:56:59,267 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=13f5a923b0a0364586ae63fe46bf7293, UNASSIGN in 2.5870 sec 2023-03-27 04:56:59,268 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893019268"}]},"ts":"1679893019268"} 2023-03-27 04:56:59,269 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-27 04:56:59,269 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-27 04:56:59,271 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.6020 sec 2023-03-27 04:57:00,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-27 04:57:00,782 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-27 04:57:00,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-27 04:57:00,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:00,787 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:00,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-27 04:57:00,788 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:00,795 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:57:00,799 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/recovered.edits] 2023-03-27 04:57:00,804 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/173274d349f84616a512d1e8fe8b47e3 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/173274d349f84616a512d1e8fe8b47e3 2023-03-27 04:57:00,806 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/1f2b941e1bf04a94bcf9d33a8c37c6db to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/1f2b941e1bf04a94bcf9d33a8c37c6db 2023-03-27 04:57:00,808 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/3f2e27212be94050a2e29cc3566461c9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/3f2e27212be94050a2e29cc3566461c9 2023-03-27 04:57:00,811 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/55650ad4c8b34e918f14497da93c717f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/55650ad4c8b34e918f14497da93c717f 2023-03-27 04:57:00,813 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/8e010ebba070429bbd51064089fce9ce to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/8e010ebba070429bbd51064089fce9ce 2023-03-27 04:57:00,815 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/a6100654eda14442beb5ce2d21546cba to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/a6100654eda14442beb5ce2d21546cba 2023-03-27 04:57:00,817 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/a65a03bd380540d68b225c699afcb780 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/a65a03bd380540d68b225c699afcb780 2023-03-27 04:57:00,819 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/cc0fc91b3b4146559b98065c9ce90080 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/cc0fc91b3b4146559b98065c9ce90080 2023-03-27 04:57:00,822 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/d70b2ae64f04472a95a9794750a58cc9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/A/d70b2ae64f04472a95a9794750a58cc9 2023-03-27 04:57:00,827 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/0bcff1d73a7a4a1db3d09241a31767d2 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/0bcff1d73a7a4a1db3d09241a31767d2 2023-03-27 04:57:00,829 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/105f40e521134403ad0218b03b77ecf3 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/105f40e521134403ad0218b03b77ecf3 2023-03-27 04:57:00,831 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/79df079dc70a482f9b155d518d3e9909 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/79df079dc70a482f9b155d518d3e9909 2023-03-27 04:57:00,832 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/807804a2cf4a4eae9df1a9f235056ef9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/807804a2cf4a4eae9df1a9f235056ef9 2023-03-27 04:57:00,834 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/8812208700154093a3cf1f84533def2a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/8812208700154093a3cf1f84533def2a 2023-03-27 04:57:00,835 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/ab47076d95044ea48ac1f94f614475a1 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/ab47076d95044ea48ac1f94f614475a1 2023-03-27 04:57:00,836 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/bf3805566c514c6f8071315e5889737a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/bf3805566c514c6f8071315e5889737a 2023-03-27 04:57:00,838 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/c326acc5e5d74678ac59f718f01a1f38 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/B/c326acc5e5d74678ac59f718f01a1f38 2023-03-27 04:57:00,841 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/162546b12efb4617a463709c23d16965 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/162546b12efb4617a463709c23d16965 2023-03-27 04:57:00,842 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/1f22a923ad5445b0ba5e1e0e4d1340c0 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/1f22a923ad5445b0ba5e1e0e4d1340c0 2023-03-27 04:57:00,843 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/755644e4db294c3da7774f27054d0a9b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/755644e4db294c3da7774f27054d0a9b 2023-03-27 04:57:00,844 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/955c3ded156441b6817711ff2052b6ee to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/955c3ded156441b6817711ff2052b6ee 2023-03-27 04:57:00,845 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/988d25dd894f4e59b644ca5526412fe4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/988d25dd894f4e59b644ca5526412fe4 2023-03-27 04:57:00,846 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/d7bcd1b394734be3b4b42b430b792088 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/d7bcd1b394734be3b4b42b430b792088 2023-03-27 04:57:00,847 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/d8dc65406ccf4d2dbaa31023866b7ff8 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/d8dc65406ccf4d2dbaa31023866b7ff8 2023-03-27 04:57:00,847 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/fc448cfd1ef84937baeed8e8fa745ffb to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/C/fc448cfd1ef84937baeed8e8fa745ffb 2023-03-27 04:57:00,849 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/recovered.edits/909.seqid to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293/recovered.edits/909.seqid 2023-03-27 04:57:00,850 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/13f5a923b0a0364586ae63fe46bf7293 2023-03-27 04:57:00,850 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-27 04:57:00,850 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-27 04:57:00,851 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-27 04:57:00,854 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303271293a4742b9a4e92963ec341ec0513d6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303271293a4742b9a4e92963ec341ec0513d6 2023-03-27 04:57:00,855 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327176185de8a5746a882298b6c3ee2395f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327176185de8a5746a882298b6c3ee2395f 2023-03-27 04:57:00,855 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303272d07b49cda174e97bc4f09d8362a0bcf to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303272d07b49cda174e97bc4f09d8362a0bcf 2023-03-27 04:57:00,856 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032733e4d037e6c44c0dbe37f37c837a71d2 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032733e4d037e6c44c0dbe37f37c837a71d2 2023-03-27 04:57:00,857 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327369d990ab29846899d5fcc83e9beba65 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327369d990ab29846899d5fcc83e9beba65 2023-03-27 04:57:00,858 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303273f4c1f698a844b8b86f4150f6a3674f6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303273f4c1f698a844b8b86f4150f6a3674f6 2023-03-27 04:57:00,859 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032740eab22e00d8473bbb5339e05f5a6b01 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032740eab22e00d8473bbb5339e05f5a6b01 2023-03-27 04:57:00,859 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032741a1b36e98224ecea9bc881242dc51c8 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032741a1b36e98224ecea9bc881242dc51c8 2023-03-27 04:57:00,860 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327450b20a5296c4364a81a32c851cc1418 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327450b20a5296c4364a81a32c851cc1418 2023-03-27 04:57:00,861 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303275cc5b1df8f5540978a00b9022057beba to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303275cc5b1df8f5540978a00b9022057beba 2023-03-27 04:57:00,862 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303278619a3084e704d718dbaa492f3bc1e47 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303278619a3084e704d718dbaa492f3bc1e47 2023-03-27 04:57:00,863 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303278e3b2f1b373b4d54b2c5a32c7c14552d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303278e3b2f1b373b4d54b2c5a32c7c14552d 2023-03-27 04:57:00,864 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032797205f1b46d643418456e5e9e9b2ce64 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032797205f1b46d643418456e5e9e9b2ce64 2023-03-27 04:57:00,865 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327a0b333ea160f44fdacc6b74756f71801 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327a0b333ea160f44fdacc6b74756f71801 2023-03-27 04:57:00,866 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327c41f48b84e3e47efafe4975f525496de to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327c41f48b84e3e47efafe4975f525496de 2023-03-27 04:57:00,866 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327c4b1eaecd6d043b99d64447f21433b2c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327c4b1eaecd6d043b99d64447f21433b2c 2023-03-27 04:57:00,867 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327cc7368eabc894b709b286a269685cf82 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327cc7368eabc894b709b286a269685cf82 2023-03-27 04:57:00,868 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327d80399e775c747b2b652373bbf6f600c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327d80399e775c747b2b652373bbf6f600c 2023-03-27 04:57:00,869 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e164065fd4a4403b880690842b89c039 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e164065fd4a4403b880690842b89c039 2023-03-27 04:57:00,870 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e6f961be87d3467296ec9f69e6d90be4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e6f961be87d3467296ec9f69e6d90be4 2023-03-27 04:57:00,873 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e8b711281f1849a1b980964fca43f345 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e8b711281f1849a1b980964fca43f345 2023-03-27 04:57:00,873 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-27 04:57:00,876 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:00,878 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-27 04:57:00,880 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-27 04:57:00,881 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:00,881 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-27 04:57:00,881 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679893020881"}]},"ts":"9223372036854775807"} 2023-03-27 04:57:00,883 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-27 04:57:00,883 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 13f5a923b0a0364586ae63fe46bf7293, NAME => 'TestAcidGuarantees,,1679892991965.13f5a923b0a0364586ae63fe46bf7293.', STARTKEY => '', ENDKEY => ''}] 2023-03-27 04:57:00,883 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-27 04:57:00,883 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679893020883"}]},"ts":"9223372036854775807"} 2023-03-27 04:57:00,885 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-27 04:57:00,886 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:00,887 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 104 msec 2023-03-27 04:57:00,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-27 04:57:00,889 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-27 04:57:00,900 INFO [Listener at localhost.localdomain/43311] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=227 (was 230), OpenFileDescriptor=561 (was 524) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=458 (was 533), ProcessCount=167 (was 167), AvailableMemoryMB=896 (was 1026) 2023-03-27 04:57:00,909 INFO [Listener at localhost.localdomain/43311] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227, OpenFileDescriptor=561, MaxFileDescriptor=60000, SystemLoadAverage=458, ProcessCount=167, AvailableMemoryMB=895 2023-03-27 04:57:00,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-27 04:57:00,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-27 04:57:00,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:00,913 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-27 04:57:00,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-27 04:57:00,914 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-27 04:57:00,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-27 04:57:00,915 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:00,915 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1 empty. 2023-03-27 04:57:00,916 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:00,916 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-27 04:57:01,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-27 04:57:01,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-27 04:57:01,323 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-27 04:57:01,325 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => c239f1ca264c8e6332ec629bacfc12b1, NAME => 'TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp 2023-03-27 04:57:01,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-27 04:57:01,733 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:57:01,734 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing c239f1ca264c8e6332ec629bacfc12b1, disabling compactions & flushes 2023-03-27 04:57:01,734 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:01,734 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:01,734 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. after waiting 0 ms 2023-03-27 04:57:01,734 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:01,734 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:01,734 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:01,741 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-27 04:57:01,742 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893021742"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679893021742"}]},"ts":"1679893021742"} 2023-03-27 04:57:01,743 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-27 04:57:01,744 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-27 04:57:01,744 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893021744"}]},"ts":"1679893021744"} 2023-03-27 04:57:01,745 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-27 04:57:01,747 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c239f1ca264c8e6332ec629bacfc12b1, ASSIGN}] 2023-03-27 04:57:01,749 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c239f1ca264c8e6332ec629bacfc12b1, ASSIGN 2023-03-27 04:57:01,750 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c239f1ca264c8e6332ec629bacfc12b1, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,44923,1679892955505; forceNewPlan=false, retain=false 2023-03-27 04:57:01,901 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=c239f1ca264c8e6332ec629bacfc12b1, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:01,902 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893021901"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679893021901"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679893021901"}]},"ts":"1679893021901"} 2023-03-27 04:57:01,904 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:57:02,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-27 04:57:02,062 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:02,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c239f1ca264c8e6332ec629bacfc12b1, NAME => 'TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.', STARTKEY => '', ENDKEY => ''} 2023-03-27 04:57:02,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:02,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:57:02,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:02,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:02,064 INFO [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:02,067 DEBUG [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A 2023-03-27 04:57:02,067 DEBUG [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A 2023-03-27 04:57:02,067 INFO [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:57:02,068 INFO [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c239f1ca264c8e6332ec629bacfc12b1 columnFamilyName A 2023-03-27 04:57:02,068 INFO [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] regionserver.HStore(310): Store=c239f1ca264c8e6332ec629bacfc12b1/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:57:02,068 INFO [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:02,070 DEBUG [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B 2023-03-27 04:57:02,070 DEBUG [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B 2023-03-27 04:57:02,070 INFO [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:57:02,070 INFO [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c239f1ca264c8e6332ec629bacfc12b1 columnFamilyName B 2023-03-27 04:57:02,071 INFO [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] regionserver.HStore(310): Store=c239f1ca264c8e6332ec629bacfc12b1/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:57:02,071 INFO [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:02,073 DEBUG [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C 2023-03-27 04:57:02,073 DEBUG [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C 2023-03-27 04:57:02,073 INFO [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:57:02,073 INFO [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c239f1ca264c8e6332ec629bacfc12b1 columnFamilyName C 2023-03-27 04:57:02,074 INFO [StoreOpener-c239f1ca264c8e6332ec629bacfc12b1-1] regionserver.HStore(310): Store=c239f1ca264c8e6332ec629bacfc12b1/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:57:02,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:02,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:02,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:02,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-27 04:57:02,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:02,082 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-27 04:57:02,083 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened c239f1ca264c8e6332ec629bacfc12b1; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70085171, jitterRate=0.044350430369377136}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-27 04:57:02,083 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:02,084 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., pid=30, masterSystemTime=1679893022057 2023-03-27 04:57:02,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:02,086 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:02,086 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=c239f1ca264c8e6332ec629bacfc12b1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:02,086 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893022086"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679893022086"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679893022086"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679893022086"}]},"ts":"1679893022086"} 2023-03-27 04:57:02,090 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-27 04:57:02,090 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 in 184 msec 2023-03-27 04:57:02,092 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-27 04:57:02,092 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c239f1ca264c8e6332ec629bacfc12b1, ASSIGN in 343 msec 2023-03-27 04:57:02,093 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-27 04:57:02,093 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893022093"}]},"ts":"1679893022093"} 2023-03-27 04:57:02,094 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-27 04:57:02,097 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-27 04:57:02,099 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1860 sec 2023-03-27 04:57:03,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-27 04:57:03,022 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-27 04:57:03,024 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x15a0890b to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:03,031 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7117e4df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:03,033 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:03,034 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42514, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:03,036 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-27 04:57:03,038 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38506, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-27 04:57:03,040 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x4d3f4b11 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:03,045 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c466b6c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:03,046 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x7187fcf3 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:03,053 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46d74611, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:03,054 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x712d78ca to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:03,059 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f837278, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:03,059 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x407b9cb0 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:03,062 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a8a575d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:03,062 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x7f456fa2 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:03,066 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6347f2e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:03,067 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x36523181 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:03,071 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ade356f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:03,071 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x6b458e37 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:03,075 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cbc74e7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:03,075 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x7bae1a64 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:03,079 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c5d1710, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:03,079 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x5654f8ca to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:03,083 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d22a55e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:03,083 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x509adaa8 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:03,086 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2179ead9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:03,091 DEBUG [hconnection-0x15147205-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:03,091 DEBUG [hconnection-0x7296065-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:03,092 DEBUG [hconnection-0x6000d087-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:03,093 DEBUG [hconnection-0x2c641cce-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:03,094 DEBUG [hconnection-0x592f87ab-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:03,094 DEBUG [hconnection-0xb565163-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:03,095 DEBUG [hconnection-0x3c5b0eb3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:03,095 DEBUG [hconnection-0x7925fd91-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:03,095 DEBUG [hconnection-0x25e43d30-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:03,097 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42526, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:03,098 DEBUG [hconnection-0x5cc2f627-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:03,097 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42536, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:03,106 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42556, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:03,106 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42562, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:03,106 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42534, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:03,106 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42576, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:03,106 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42592, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:03,108 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42594, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:03,112 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42590, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:03,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:03,112 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42610, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:03,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:03,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:03,117 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:03,117 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:03,117 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:03,117 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:03,118 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:03,118 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:03,122 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:03,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:03,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:03,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:03,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:03,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:03,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:03,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:03,123 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,123 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:03,123 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:03,123 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:03,123 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:03,124 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:03,124 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:03,126 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:03,126 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:03,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:03,128 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:57:03,128 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:03,131 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:03,131 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:03,131 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:03,131 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:03,131 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:03,131 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:03,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893083147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1679893023145, max timestamp=1679893023151Num uniques 859; 2023-03-27 04:57:03,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-27 04:57:03,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679893023145, max timestamp=1679893023151Num uniques 859; 2023-03-27 04:57:03,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-27 04:57:03,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893023146, max timestamp=1679893023151Num uniques 859; 2023-03-27 04:57:03,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:57:03,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893083167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893083170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893083171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893083172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893083255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893083273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893083275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893083275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893083275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893083458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893083477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893083477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893083477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893083479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,557 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/f3601d643cf14e038ec451d93ced9d11 2023-03-27 04:57:03,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893083762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893083779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893083781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893083782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:03,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893083782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:03,970 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/2b78d509c97845c6a9fb76d825e50b22 2023-03-27 04:57:04,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:04,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893084264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:04,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893084284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:04,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893084285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:04,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893084286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:04,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893084288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,386 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/4c06ce6a33dc4399abbde618e62d74dc 2023-03-27 04:57:04,391 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/f3601d643cf14e038ec451d93ced9d11 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/f3601d643cf14e038ec451d93ced9d11 2023-03-27 04:57:04,395 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/f3601d643cf14e038ec451d93ced9d11, entries=150, sequenceid=13, filesize=11.6 K 2023-03-27 04:57:04,396 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/2b78d509c97845c6a9fb76d825e50b22 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/2b78d509c97845c6a9fb76d825e50b22 2023-03-27 04:57:04,399 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/2b78d509c97845c6a9fb76d825e50b22, entries=150, sequenceid=13, filesize=11.6 K 2023-03-27 04:57:04,399 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/4c06ce6a33dc4399abbde618e62d74dc as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/4c06ce6a33dc4399abbde618e62d74dc 2023-03-27 04:57:04,402 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/4c06ce6a33dc4399abbde618e62d74dc, entries=150, sequenceid=13, filesize=11.6 K 2023-03-27 04:57:04,403 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=123.00 KB/125950 for c239f1ca264c8e6332ec629bacfc12b1 in 1276ms, sequenceid=13, compaction requested=false 2023-03-27 04:57:04,403 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:04,403 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:04,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:04,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:04,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:04,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:04,404 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,404 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:04,405 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:04,405 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:04,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:04,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dc7058a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:04,406 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:04,406 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:04,406 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:04,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:04,407 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:04,407 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:04,407 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:04,407 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,407 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:04,407 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:04,407 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:04,407 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,407 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:04,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:04,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:04,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:04,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:04,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:04,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:04,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:04,409 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:04,409 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:04,410 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:04,410 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:04,410 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:04,411 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:04,411 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:04,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:04,411 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:04,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:04,411 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:04,412 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,414 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,414 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:04,414 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:04,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:04,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:04,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:04,414 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:04,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:04,414 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:04,414 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:04,414 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:04,414 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:04,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:04,414 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,414 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:04,414 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:04,414 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:04,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:04,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:04,514 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:04,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:04,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:04,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:04,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:04,517 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:04,517 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:04,517 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:04,517 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:04,518 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:04,518 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:04,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:04,518 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:04,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:04,519 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:04,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:04,519 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:04,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:04,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:04,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:04,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:04,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:04,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:04,520 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:04,520 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:04,520 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:04,520 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:04,520 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:04,520 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:04,520 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:04,520 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=123.00 KB heapSize=340.84 KB 2023-03-27 04:57:04,520 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:04,520 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:04,521 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:04,521 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:04,521 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:04,521 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:04,528 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/1db194bfa3404a0d9e4a782ffa5a2c1b 2023-03-27 04:57:04,807 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-27 04:57:04,956 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/916d986e286b451db619194a79c9711d 2023-03-27 04:57:05,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:05,289 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:05,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893085300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893085301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893085301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893085302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893085302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,364 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/ffa1842cf6c74d08ac909b7244465136 2023-03-27 04:57:05,368 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/1db194bfa3404a0d9e4a782ffa5a2c1b as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/1db194bfa3404a0d9e4a782ffa5a2c1b 2023-03-27 04:57:05,371 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/1db194bfa3404a0d9e4a782ffa5a2c1b, entries=150, sequenceid=46, filesize=11.6 K 2023-03-27 04:57:05,372 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/916d986e286b451db619194a79c9711d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/916d986e286b451db619194a79c9711d 2023-03-27 04:57:05,376 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/916d986e286b451db619194a79c9711d, entries=150, sequenceid=46, filesize=11.6 K 2023-03-27 04:57:05,377 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/ffa1842cf6c74d08ac909b7244465136 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/ffa1842cf6c74d08ac909b7244465136 2023-03-27 04:57:05,381 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/ffa1842cf6c74d08ac909b7244465136, entries=150, sequenceid=46, filesize=11.6 K 2023-03-27 04:57:05,382 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~123.00 KB/125950, heapSize ~340.80 KB/348976, currentSize=67.09 KB/68700 for c239f1ca264c8e6332ec629bacfc12b1 in 862ms, sequenceid=46, compaction requested=false 2023-03-27 04:57:05,382 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:05,382 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:05,382 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:05,382 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:05,382 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,382 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:05,382 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:05,383 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,383 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,383 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,384 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:05,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:05,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@467aa325[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:05,385 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:05,385 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,386 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,386 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:05,386 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:05,386 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:05,386 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,386 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:05,387 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:05,387 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:05,387 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,387 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:05,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:05,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:05,389 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:05,389 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:05,389 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:05,389 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:05,389 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:05,391 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,391 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,391 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:05,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:05,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:05,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,392 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,393 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,395 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,395 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:05,395 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:05,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:05,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:05,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:05,395 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:05,395 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,395 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,395 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,395 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,396 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,395 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:05,395 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:05,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:05,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:05,396 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:05,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:05,406 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-27 04:57:05,407 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:05,407 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:05,407 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:05,407 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:05,407 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:05,407 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:05,414 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/a5bff4b34f1a45fab83e14d45abd074a 2023-03-27 04:57:05,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893085414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893085415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893085415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893085415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893085416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,496 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:05,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:05,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:05,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:05,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:05,499 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:05,499 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:05,499 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:05,499 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:05,499 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,499 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,500 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:05,500 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:05,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:05,500 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,500 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:05,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:05,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:05,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:05,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:05,501 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:05,501 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:05,501 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:05,502 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:05,502 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:05,502 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:05,502 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:05,502 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:05,502 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:05,502 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:05,502 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:05,502 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,502 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:05,502 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:05,503 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,503 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,503 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,503 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:05,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:05,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b0565ab[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:05,504 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:05,504 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,505 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,505 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:05,505 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:05,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:05,505 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:05,506 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:05,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,506 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,506 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:05,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:05,508 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:05,508 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:05,508 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:05,508 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:05,508 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:05,509 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,509 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,509 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:05,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:05,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:05,510 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,511 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:05,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:05,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:05,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:05,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:05,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,512 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:05,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,512 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:05,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:05,513 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:05,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:05,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:05,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:05,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893085518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893085518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893085519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893085519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893085520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,613 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:05,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:05,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:05,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:05,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:05,616 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:05,616 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:05,616 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:05,616 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:05,616 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,616 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,617 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:05,617 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:05,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:05,617 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,617 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:05,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:05,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:05,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:05,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:05,618 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:05,618 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:05,618 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:05,620 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:05,620 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:05,620 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:05,620 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:05,620 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:05,620 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:05,620 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:05,620 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:05,620 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,620 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:05,620 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:05,621 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,621 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,621 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,621 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:05,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:05,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@741de846[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:05,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:05,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,623 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,623 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:05,623 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:05,623 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:05,623 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,623 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:05,624 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:05,624 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,624 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:05,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,624 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:05,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:05,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:05,625 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:05,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:05,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:05,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:05,626 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:05,626 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,626 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,626 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:05,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,627 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:05,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:05,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,628 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:05,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:05,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:05,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:05,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:05,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:05,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:05,629 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:05,630 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:05,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,630 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:05,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:05,630 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:05,630 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,630 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,630 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,630 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893085721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893085722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893085722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893085727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:05,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893085727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,730 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:05,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:05,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:05,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:05,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:05,732 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:05,732 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:05,733 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:05,733 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:05,733 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,733 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,734 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:05,734 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:05,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:05,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,734 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,734 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:05,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:05,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:05,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:05,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:05,735 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:05,735 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:05,735 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:05,736 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:05,736 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:05,736 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:05,736 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:05,736 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:05,736 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:05,736 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:05,736 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:05,736 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,736 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:05,736 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:05,737 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,737 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,737 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,737 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:05,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2123581a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:05,738 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:05,739 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,739 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,739 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:05,739 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,739 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:05,739 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:05,739 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:05,740 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:05,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:05,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:05,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:05,742 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:05,742 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:05,742 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:05,742 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:05,742 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:05,743 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,743 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,744 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:05,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:05,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:05,745 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,745 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,747 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,747 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,747 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,747 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:05,747 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:05,747 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:05,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:05,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:05,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:05,747 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:05,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:05,747 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,747 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,747 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:05,747 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,747 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:05,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:05,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:05,833 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/1f92995c526e4806ad60265a35be82ca 2023-03-27 04:57:05,847 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:05,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:05,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:05,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:05,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:05,862 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:05,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:05,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:05,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:05,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,863 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:05,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:05,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:05,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:05,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:05,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:05,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:05,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:05,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:05,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:05,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:05,864 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:05,864 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:05,864 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:05,864 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:05,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:05,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:05,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:05,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:05,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,866 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,866 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:05,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:05,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32d3a490[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:05,867 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:05,867 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,867 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:05,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:05,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:05,868 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,868 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:05,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:05,868 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:05,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:05,870 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:05,871 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:05,871 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:05,871 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:05,871 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:05,871 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,872 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,872 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:05,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,874 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,875 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:05,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:05,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:05,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:05,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,876 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,876 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,876 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:05,876 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:05,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:05,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:05,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:05,976 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:05,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:05,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:05,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:05,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:05,979 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:05,979 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:05,979 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:05,979 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:05,979 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,979 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,980 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:05,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:05,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:05,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:05,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:05,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:05,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:05,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:05,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:05,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:05,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:05,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:05,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:05,981 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:05,981 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:05,981 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:05,981 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:05,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:05,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:05,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:05,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:05,982 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,982 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,982 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,982 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:05,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@125bb75c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:05,983 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:05,983 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,984 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:05,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:05,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:05,984 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,984 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:05,985 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,985 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:05,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,985 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:05,985 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:05,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:05,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:05,987 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:05,987 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:05,987 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:05,987 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:05,987 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:05,987 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,987 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,988 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:05,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:05,988 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,989 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:05,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,990 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,990 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:05,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:05,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:05,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:05,990 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,990 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:05,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:05,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:05,990 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:05,991 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:05,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:05,991 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:05,991 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:05,991 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:05,991 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:05,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:05,991 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893086025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893086026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893086027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893086028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893086031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,091 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:06,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:06,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:06,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:06,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:06,094 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:06,094 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:06,094 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:06,094 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:06,094 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,094 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:06,095 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:06,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:06,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:06,095 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,095 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,095 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:06,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:06,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:06,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:06,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:06,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:06,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:06,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:06,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:06,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:06,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:06,106 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:06,106 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:06,106 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:06,106 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:06,106 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:06,106 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:06,106 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,106 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:06,106 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:06,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:06,111 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:06,112 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:06,112 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:06,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:06,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:06,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:06,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:06,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:06,113 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:06,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@99f2289[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:06,113 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:06,116 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:06,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:06,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:06,116 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:06,116 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:06,116 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:06,117 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,117 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:06,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,117 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:06,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:06,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:06,117 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:06,117 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:06,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:06,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:06,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:06,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:06,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:06,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:06,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:06,119 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:06,119 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:06,119 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:06,119 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:06,119 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:06,120 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,120 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,120 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:06,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:06,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:06,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:06,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:06,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:06,121 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:06,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:06,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:06,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:06,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:06,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,124 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,124 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:06,124 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:06,124 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:06,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:06,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:06,124 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:06,124 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:06,124 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:06,124 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:06,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:06,124 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:06,125 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:06,124 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:06,125 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:06,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:06,225 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:06,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:06,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:06,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:06,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:06,227 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:06,227 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:06,227 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:06,227 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:06,228 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,228 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:06,228 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,229 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:06,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:06,229 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,229 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:06,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:06,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:06,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:06,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:06,230 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:06,230 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:06,230 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:06,230 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:06,231 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,231 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:06,231 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:06,231 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:06,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:06,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32c74828[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:06,232 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:06,232 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:06,233 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:06,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:06,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:06,233 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,233 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:06,233 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:06,233 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:06,233 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:06,234 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,234 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:06,234 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:06,234 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:06,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:06,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:06,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:06,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:06,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:06,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:06,236 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:06,236 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:06,236 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:06,236 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:06,236 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:06,237 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,237 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,237 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:06,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:06,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:06,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:06,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:06,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:06,237 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:06,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:06,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:06,238 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:06,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:06,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,240 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,240 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:06,240 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:06,240 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:06,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:06,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:06,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:06,240 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:06,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:06,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:06,240 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:06,240 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,240 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:06,240 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:06,240 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:06,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:06,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:06,240 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:06,240 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,262 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/953c0410b9944b1bbd3ff3ed2423e2f7 2023-03-27 04:57:06,265 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/a5bff4b34f1a45fab83e14d45abd074a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/a5bff4b34f1a45fab83e14d45abd074a 2023-03-27 04:57:06,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/a5bff4b34f1a45fab83e14d45abd074a, entries=150, sequenceid=62, filesize=11.6 K 2023-03-27 04:57:06,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/1f92995c526e4806ad60265a35be82ca as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/1f92995c526e4806ad60265a35be82ca 2023-03-27 04:57:06,272 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/1f92995c526e4806ad60265a35be82ca, entries=150, sequenceid=62, filesize=11.6 K 2023-03-27 04:57:06,273 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/953c0410b9944b1bbd3ff3ed2423e2f7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/953c0410b9944b1bbd3ff3ed2423e2f7 2023-03-27 04:57:06,276 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/953c0410b9944b1bbd3ff3ed2423e2f7, entries=150, sequenceid=62, filesize=11.6 K 2023-03-27 04:57:06,276 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for c239f1ca264c8e6332ec629bacfc12b1 in 870ms, sequenceid=62, compaction requested=true 2023-03-27 04:57:06,276 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:06,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:06,277 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:06,277 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:06,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:06,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:06,278 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:06,278 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/B is initiating minor compaction (all files) 2023-03-27 04:57:06,278 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:06,278 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/B in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:06,278 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/A is initiating minor compaction (all files) 2023-03-27 04:57:06,278 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/2b78d509c97845c6a9fb76d825e50b22, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/916d986e286b451db619194a79c9711d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/1f92995c526e4806ad60265a35be82ca] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=34.8 K 2023-03-27 04:57:06,278 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/A in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:06,278 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/f3601d643cf14e038ec451d93ced9d11, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/1db194bfa3404a0d9e4a782ffa5a2c1b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/a5bff4b34f1a45fab83e14d45abd074a] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=34.8 K 2023-03-27 04:57:06,278 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 2b78d509c97845c6a9fb76d825e50b22, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679893023119 2023-03-27 04:57:06,278 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting f3601d643cf14e038ec451d93ced9d11, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679893023119 2023-03-27 04:57:06,279 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 916d986e286b451db619194a79c9711d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1679893023169 2023-03-27 04:57:06,279 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 1db194bfa3404a0d9e4a782ffa5a2c1b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1679893023169 2023-03-27 04:57:06,279 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 1f92995c526e4806ad60265a35be82ca, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679893025301 2023-03-27 04:57:06,279 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting a5bff4b34f1a45fab83e14d45abd074a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679893025301 2023-03-27 04:57:06,284 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#B#compaction#214 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:06,288 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#A#compaction#215 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:06,295 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/b845f2c4122a4f6f8e669b9b49465931 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/b845f2c4122a4f6f8e669b9b49465931 2023-03-27 04:57:06,299 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/B of c239f1ca264c8e6332ec629bacfc12b1 into b845f2c4122a4f6f8e669b9b49465931(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:06,299 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:06,299 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/B, priority=13, startTime=1679893026277; duration=0sec 2023-03-27 04:57:06,299 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:06,299 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:06,300 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:06,300 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/C is initiating minor compaction (all files) 2023-03-27 04:57:06,300 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/C in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:06,301 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/4c06ce6a33dc4399abbde618e62d74dc, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/ffa1842cf6c74d08ac909b7244465136, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/953c0410b9944b1bbd3ff3ed2423e2f7] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=34.8 K 2023-03-27 04:57:06,301 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 4c06ce6a33dc4399abbde618e62d74dc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679893023119 2023-03-27 04:57:06,302 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting ffa1842cf6c74d08ac909b7244465136, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1679893023169 2023-03-27 04:57:06,302 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 953c0410b9944b1bbd3ff3ed2423e2f7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679893025301 2023-03-27 04:57:06,309 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#C#compaction#216 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:06,320 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/8637bf210c8e45cea98564c81a5761b8 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/8637bf210c8e45cea98564c81a5761b8 2023-03-27 04:57:06,325 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/C of c239f1ca264c8e6332ec629bacfc12b1 into 8637bf210c8e45cea98564c81a5761b8(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:06,326 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:06,326 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/C, priority=13, startTime=1679893026277; duration=0sec 2023-03-27 04:57:06,326 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:06,340 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:06,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:06,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:06,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:06,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:06,344 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:06,344 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:06,344 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:06,344 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:06,344 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,344 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:06,345 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:06,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:06,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:06,345 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:06,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:06,346 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,346 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:06,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:06,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:06,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:06,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:06,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:06,346 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:06,347 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:06,347 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:06,354 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:06,354 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:06,354 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:06,354 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:06,354 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-27 04:57:06,354 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:06,354 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:06,355 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:06,355 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:06,355 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:06,355 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:06,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:06,531 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:06,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893086537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893086538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893086539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893086540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893086540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893086641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893086641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893086641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893086642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893086643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,696 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/ec53091b9cec43119a2420b2b7bcf08c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ec53091b9cec43119a2420b2b7bcf08c 2023-03-27 04:57:06,699 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/A of c239f1ca264c8e6332ec629bacfc12b1 into ec53091b9cec43119a2420b2b7bcf08c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:06,699 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:06,699 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/A, priority=13, startTime=1679893026277; duration=0sec 2023-03-27 04:57:06,699 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:06,765 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/8e813c9831164932a2a1376d69ea559c 2023-03-27 04:57:06,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893086845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893086845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893086845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893086845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:06,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:06,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893086845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893087147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893087148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893087148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893087148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893087148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,173 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/2881c63986184959bc376482d796d161 2023-03-27 04:57:07,582 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/5975cb3a81d74afab8899bf00857330b 2023-03-27 04:57:07,586 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/8e813c9831164932a2a1376d69ea559c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8e813c9831164932a2a1376d69ea559c 2023-03-27 04:57:07,589 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8e813c9831164932a2a1376d69ea559c, entries=150, sequenceid=80, filesize=11.6 K 2023-03-27 04:57:07,590 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/2881c63986184959bc376482d796d161 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/2881c63986184959bc376482d796d161 2023-03-27 04:57:07,593 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/2881c63986184959bc376482d796d161, entries=150, sequenceid=80, filesize=11.6 K 2023-03-27 04:57:07,594 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/5975cb3a81d74afab8899bf00857330b as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/5975cb3a81d74afab8899bf00857330b 2023-03-27 04:57:07,597 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/5975cb3a81d74afab8899bf00857330b, entries=150, sequenceid=80, filesize=11.6 K 2023-03-27 04:57:07,598 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for c239f1ca264c8e6332ec629bacfc12b1 in 1244ms, sequenceid=80, compaction requested=false 2023-03-27 04:57:07,598 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:07,598 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:07,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:07,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:07,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:07,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:07,599 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,599 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:07,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:07,599 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,599 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:07,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:07,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:07,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:07,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:07,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d9f8dc4[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:07,601 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:07,601 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,601 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:07,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:07,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:07,601 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,602 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:07,602 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:07,602 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:07,602 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:07,602 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:07,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:07,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:07,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:07,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:07,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:07,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:07,605 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:07,605 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:07,605 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:07,605 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:07,605 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:07,606 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,606 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:07,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:07,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:07,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:07,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:07,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:07,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,608 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:07,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:07,612 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:07,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:07,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:07,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:07,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:07,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:07,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:07,620 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:07,620 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:07,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:07,620 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:07,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:07,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:07,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:07,652 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-27 04:57:07,652 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:07,652 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:07,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:07,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:07,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:07,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:07,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893087665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893087666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893087668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893087668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893087668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,720 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:07,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:07,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:07,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:07,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:07,723 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:07,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:07,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:07,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:07,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:07,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:07,725 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:07,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:07,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:07,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:07,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:07,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:07,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:07,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:07,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:07,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:07,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:07,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:07,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:07,728 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:07,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:07,728 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:07,728 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:07,728 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:07,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:07,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:07,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:07,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:07,729 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,729 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:07,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:07,729 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,729 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:07,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:07,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:07,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:07,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:07,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5231387c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:07,731 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:07,731 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,731 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,731 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:07,731 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:07,731 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,731 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:07,731 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:07,732 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:07,732 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,732 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:07,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:07,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:07,732 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:07,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:07,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:07,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:07,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:07,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:07,734 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:07,734 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:07,734 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:07,734 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:07,734 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:07,735 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,735 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,735 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:07,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:07,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:07,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:07,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:07,736 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:07,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:07,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,736 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:07,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,738 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:07,738 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:07,738 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:07,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:07,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:07,738 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,738 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:07,738 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:07,738 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,738 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,738 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,738 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:07,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:07,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:07,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:07,739 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:07,739 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:07,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:07,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893087769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893087769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893087772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893087772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893087772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,839 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:07,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:07,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:07,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:07,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:07,841 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:07,842 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:07,842 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:07,842 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:07,842 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,842 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:07,843 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,843 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:07,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:07,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:07,843 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,843 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:07,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:07,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:07,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:07,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:07,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:07,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:07,844 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:07,844 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:07,844 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:07,844 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:07,844 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:07,844 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:07,844 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:07,844 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:07,844 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:07,844 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:07,844 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:07,844 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,844 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:07,844 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:07,846 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,846 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:07,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:07,846 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,846 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:07,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:07,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:07,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:07,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:07,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@224ad3d9[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:07,848 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:07,848 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,848 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,848 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:07,848 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:07,848 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:07,848 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,849 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:07,849 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:07,849 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,849 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:07,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:07,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:07,849 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:07,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:07,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:07,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:07,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:07,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:07,851 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:07,851 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:07,851 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:07,851 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:07,851 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:07,852 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,852 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,852 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:07,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:07,853 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:07,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:07,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,855 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:07,855 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:07,855 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:07,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:07,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:07,855 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:07,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:07,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:07,855 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:07,856 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:07,856 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:07,856 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:07,856 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,856 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,856 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,856 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:07,956 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:07,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:07,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:07,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:07,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:07,959 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:07,959 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:07,959 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:07,959 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:07,959 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,959 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:07,960 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:07,960 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:07,960 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:07,960 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:07,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:07,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:07,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:07,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:07,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:07,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:07,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:07,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:07,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:07,961 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:07,961 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:07,961 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:07,961 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:07,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:07,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:07,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:07,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:07,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,962 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:07,962 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,962 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:07,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:07,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:07,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:07,963 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:07,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a4449b0[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:07,963 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,964 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,964 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:07,964 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:07,964 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:07,964 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,964 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:07,965 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,965 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:07,965 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:07,965 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:07,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:07,967 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:07,967 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:07,967 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:07,967 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:07,967 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:07,968 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:07,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:07,968 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,968 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:07,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:07,968 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:07,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:07,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:07,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,970 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:07,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:07,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,971 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,971 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:07,971 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:07,971 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:07,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:07,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:07,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:07,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:07,971 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:07,971 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:07,971 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:07,971 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:07,971 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,971 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:07,972 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:07,972 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:07,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:07,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:07,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893087972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893087972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893087976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893087976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:07,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:07,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893087976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,066 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/ddad6ec5c1194d398382544242552fad 2023-03-27 04:57:08,072 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:08,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:08,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:08,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:08,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:08,075 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:08,075 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:08,075 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:08,075 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:08,075 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,075 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:08,076 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:08,076 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,076 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:08,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:08,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:08,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:08,085 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:08,085 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:08,085 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:08,085 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:08,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:08,086 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:08,086 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:08,086 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:08,086 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:08,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:08,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:08,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:08,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:08,091 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,091 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:08,092 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,092 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:08,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:08,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b541091[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:08,093 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:08,093 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,094 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,095 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:08,095 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:08,095 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,095 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:08,095 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:08,095 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:08,095 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,095 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:08,096 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:08,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:08,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:08,098 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:08,098 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:08,098 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:08,098 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:08,098 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:08,099 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,099 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:08,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,099 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:08,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:08,105 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:08,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:08,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,111 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,111 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,111 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,111 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:08,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:08,111 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:08,111 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:08,111 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:08,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:08,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:08,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:08,112 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:08,112 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:08,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:08,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:08,212 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:08,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:08,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:08,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:08,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:08,218 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:08,218 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:08,218 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:08,218 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:08,219 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,219 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,219 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:08,219 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:08,220 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,220 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:08,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:08,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:08,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:08,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:08,229 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:08,229 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:08,229 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:08,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:08,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:08,230 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:08,230 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:08,230 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:08,230 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:08,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:08,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:08,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:08,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:08,233 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,233 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:08,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:08,233 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,233 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:08,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:08,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:08,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:08,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:08,235 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:08,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@107d235f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:08,235 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,236 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,236 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:08,236 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:08,236 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:08,236 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,236 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:08,237 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:08,237 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:08,237 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,237 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:08,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:08,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:08,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:08,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:08,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:08,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:08,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:08,239 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:08,239 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:08,239 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:08,239 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:08,239 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:08,240 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,240 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:08,240 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:08,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:08,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:08,241 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,242 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:08,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,244 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:08,244 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:08,244 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,244 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:08,244 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:08,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:08,244 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,244 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,244 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,244 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,244 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:08,244 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:08,244 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:08,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:08,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:08,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:08,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:08,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:08,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893088274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:08,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893088275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:08,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893088278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:08,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893088280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:08,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893088281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,344 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:08,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:08,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:08,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:08,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:08,348 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:08,348 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:08,348 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:08,348 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:08,348 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,348 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,349 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:08,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:08,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:08,349 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,349 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,349 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:08,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:08,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:08,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:08,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:08,351 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:08,351 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:08,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:08,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:08,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:08,352 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:08,352 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:08,352 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:08,352 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:08,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:08,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:08,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:08,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:08,354 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,354 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:08,354 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:08,354 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:08,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:08,355 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:08,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@645dee92[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:08,355 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,356 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,356 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,356 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:08,356 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:08,356 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:08,356 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:08,357 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,357 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:08,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:08,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:08,357 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:08,357 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:08,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:08,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:08,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:08,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:08,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:08,358 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:08,358 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:08,359 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:08,359 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:08,359 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:08,359 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,359 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:08,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,359 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:08,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:08,360 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:08,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:08,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:08,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:08,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,361 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:08,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,363 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,363 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,363 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,363 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:08,363 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:08,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:08,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:08,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:08,363 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:08,363 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,363 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,363 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,363 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:08,363 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:08,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:08,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:08,363 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:08,463 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:08,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:08,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:08,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:08,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:08,466 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:08,466 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:08,466 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:08,466 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:08,467 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,467 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,467 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:08,467 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:08,467 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,467 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:08,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:08,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:08,468 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:08,468 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:08,468 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:08,468 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:08,469 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:08,469 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:08,469 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:08,469 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:08,469 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:08,469 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:08,469 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:08,469 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,469 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:08,469 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:08,470 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,470 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:08,470 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:08,470 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:08,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:08,471 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:08,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c5c3260[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:08,471 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,472 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,472 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:08,472 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:08,472 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,472 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:08,472 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:08,473 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:08,473 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:08,473 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:08,473 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:08,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:08,474 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:08,475 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:08,475 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:08,475 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:08,475 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:08,475 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/db75863e346a4a5b97d175e0af356831 2023-03-27 04:57:08,475 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,476 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:08,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,476 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:08,476 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,478 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,480 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:08,480 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:08,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:08,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:08,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:08,480 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,480 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:08,481 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,481 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,481 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:08,481 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:08,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:08,481 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,481 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,481 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:08,481 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:08,486 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/260ee3377a6049649d69967a50e17c4f 2023-03-27 04:57:08,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/ddad6ec5c1194d398382544242552fad as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ddad6ec5c1194d398382544242552fad 2023-03-27 04:57:08,493 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ddad6ec5c1194d398382544242552fad, entries=150, sequenceid=99, filesize=11.6 K 2023-03-27 04:57:08,494 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/db75863e346a4a5b97d175e0af356831 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/db75863e346a4a5b97d175e0af356831 2023-03-27 04:57:08,497 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/db75863e346a4a5b97d175e0af356831, entries=150, sequenceid=99, filesize=11.6 K 2023-03-27 04:57:08,497 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/260ee3377a6049649d69967a50e17c4f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/260ee3377a6049649d69967a50e17c4f 2023-03-27 04:57:08,500 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/260ee3377a6049649d69967a50e17c4f, entries=150, sequenceid=99, filesize=11.6 K 2023-03-27 04:57:08,500 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=87.22 KB/89310 for c239f1ca264c8e6332ec629bacfc12b1 in 848ms, sequenceid=99, compaction requested=true 2023-03-27 04:57:08,500 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:08,500 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:08,500 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:08,501 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:08,500 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:08,501 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:08,501 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:08,501 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:08,501 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/A is initiating minor compaction (all files) 2023-03-27 04:57:08,501 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/B is initiating minor compaction (all files) 2023-03-27 04:57:08,501 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/A in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:08,501 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/B in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:08,502 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ec53091b9cec43119a2420b2b7bcf08c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8e813c9831164932a2a1376d69ea559c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ddad6ec5c1194d398382544242552fad] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=34.9 K 2023-03-27 04:57:08,502 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/b845f2c4122a4f6f8e669b9b49465931, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/2881c63986184959bc376482d796d161, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/db75863e346a4a5b97d175e0af356831] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=34.9 K 2023-03-27 04:57:08,502 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting ec53091b9cec43119a2420b2b7bcf08c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679893025301 2023-03-27 04:57:08,503 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting b845f2c4122a4f6f8e669b9b49465931, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679893025301 2023-03-27 04:57:08,503 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 8e813c9831164932a2a1376d69ea559c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1679893025413 2023-03-27 04:57:08,503 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 2881c63986184959bc376482d796d161, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1679893025413 2023-03-27 04:57:08,503 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting ddad6ec5c1194d398382544242552fad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679893026535 2023-03-27 04:57:08,503 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting db75863e346a4a5b97d175e0af356831, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679893026535 2023-03-27 04:57:08,511 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#B#compaction#224 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:08,511 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#A#compaction#223 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:08,581 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:08,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:08,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:08,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:08,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:08,583 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:08,584 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:08,584 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:08,584 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:08,584 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,584 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,585 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:08,585 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:08,585 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,585 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:08,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:08,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:08,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:08,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:08,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:08,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:08,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:08,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:08,586 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:08,586 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:08,586 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-27 04:57:08,586 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:08,587 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:08,587 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:08,587 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:08,587 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:08,587 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:08,593 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/5833f1846ebd4f0aacae66e6772b796d 2023-03-27 04:57:08,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:08,781 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:08,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:08,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893088792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:08,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893088792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:08,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893088793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:08,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:08,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893088793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893088794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:08,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893088896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:08,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893088896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:08,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893088898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:08,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893088899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:08,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893088899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:08,920 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/ddc10a0be8604f31abe8df7361fc018b as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ddc10a0be8604f31abe8df7361fc018b 2023-03-27 04:57:08,926 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/4d2f403efa67460980f892e181148204 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/4d2f403efa67460980f892e181148204 2023-03-27 04:57:08,929 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/A of c239f1ca264c8e6332ec629bacfc12b1 into ddc10a0be8604f31abe8df7361fc018b(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:08,929 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:08,929 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/A, priority=13, startTime=1679893028500; duration=0sec 2023-03-27 04:57:08,929 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:08,929 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:08,931 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:08,931 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/C is initiating minor compaction (all files) 2023-03-27 04:57:08,931 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/C in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:08,931 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/8637bf210c8e45cea98564c81a5761b8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/5975cb3a81d74afab8899bf00857330b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/260ee3377a6049649d69967a50e17c4f] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=34.9 K 2023-03-27 04:57:08,932 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 8637bf210c8e45cea98564c81a5761b8, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679893025301 2023-03-27 04:57:08,932 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 5975cb3a81d74afab8899bf00857330b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1679893025413 2023-03-27 04:57:08,932 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 260ee3377a6049649d69967a50e17c4f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679893026535 2023-03-27 04:57:08,936 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/B of c239f1ca264c8e6332ec629bacfc12b1 into 4d2f403efa67460980f892e181148204(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:08,936 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:08,936 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/B, priority=13, startTime=1679893028500; duration=0sec 2023-03-27 04:57:08,936 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:08,959 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#C#compaction#227 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:09,005 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/38d3cc61e8eb4aab915c9aeefa597ed2 2023-03-27 04:57:09,012 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/4b29091dd1dd45c69f80d506ee97ce66 2023-03-27 04:57:09,015 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/5833f1846ebd4f0aacae66e6772b796d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/5833f1846ebd4f0aacae66e6772b796d 2023-03-27 04:57:09,018 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/5833f1846ebd4f0aacae66e6772b796d, entries=150, sequenceid=115, filesize=11.6 K 2023-03-27 04:57:09,018 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/38d3cc61e8eb4aab915c9aeefa597ed2 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/38d3cc61e8eb4aab915c9aeefa597ed2 2023-03-27 04:57:09,021 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/38d3cc61e8eb4aab915c9aeefa597ed2, entries=150, sequenceid=115, filesize=11.6 K 2023-03-27 04:57:09,022 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/4b29091dd1dd45c69f80d506ee97ce66 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/4b29091dd1dd45c69f80d506ee97ce66 2023-03-27 04:57:09,025 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/4b29091dd1dd45c69f80d506ee97ce66, entries=150, sequenceid=115, filesize=11.6 K 2023-03-27 04:57:09,025 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for c239f1ca264c8e6332ec629bacfc12b1 in 439ms, sequenceid=115, compaction requested=false 2023-03-27 04:57:09,026 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:09,026 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,026 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:09,026 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:09,026 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,026 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:09,026 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:09,027 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,027 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,027 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:09,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:09,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e767e2[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:09,028 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:09,028 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,029 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,029 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:09,029 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:09,029 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:09,029 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,029 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:09,031 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,031 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,031 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:09,031 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:09,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:09,034 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:09,034 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:09,034 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:09,034 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:09,034 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:09,035 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,035 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:09,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,035 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:09,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:09,035 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,036 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,038 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,038 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,038 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:09,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:09,038 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,038 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:09,038 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:09,038 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:09,038 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,038 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:09,038 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:09,039 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:09,038 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:09,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:09,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:09,100 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-27 04:57:09,100 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:09,101 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:09,101 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:09,101 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:09,101 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:09,101 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:09,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893089108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893089109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893089109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893089110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893089110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,138 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:09,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:09,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:09,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:09,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:09,141 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:09,141 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:09,141 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:09,141 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:09,141 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,141 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,143 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,143 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:09,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:09,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:09,143 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,143 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:09,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:09,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:09,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:09,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:09,144 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:09,144 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:09,144 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:09,144 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:09,144 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:09,144 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,144 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:09,144 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:09,144 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,144 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:09,144 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:09,144 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,144 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:09,145 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:09,145 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,145 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,146 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,146 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:09,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:09,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e79cb9a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:09,146 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:09,147 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,147 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,147 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:09,147 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:09,147 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,147 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:09,147 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:09,148 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,148 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,148 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:09,148 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:09,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:09,150 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:09,150 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:09,150 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:09,150 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:09,150 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:09,151 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,151 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,151 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:09,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:09,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:09,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,151 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,152 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,154 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,154 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,154 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,154 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,154 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:09,154 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:09,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:09,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:09,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:09,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:09,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:09,154 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,154 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:09,154 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,154 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,154 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:09,154 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:09,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:09,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:09,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893089211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893089211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893089213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893089213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893089213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,254 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:09,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:09,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:09,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:09,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:09,257 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:09,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:09,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:09,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:09,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,258 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,258 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,258 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:09,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:09,258 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,258 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:09,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:09,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:09,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:09,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:09,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:09,260 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:09,260 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:09,260 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,260 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:09,260 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:09,260 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,260 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:09,260 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:09,260 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,260 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:09,260 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:09,261 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,261 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,261 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,261 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:09,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:09,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755b93b9[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:09,262 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:09,262 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,263 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,263 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,263 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:09,263 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:09,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:09,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:09,264 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:09,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:09,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:09,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:09,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:09,267 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:09,267 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:09,267 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:09,267 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:09,267 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:09,267 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,268 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:09,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,268 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,268 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,269 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,271 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:09,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:09,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:09,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:09,271 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:09,271 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:09,272 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:09,272 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:09,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:09,273 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:09,273 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,273 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,273 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,273 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,273 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,273 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:09,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:09,367 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/d3fb6c005fc04179afda01407cf399d7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/d3fb6c005fc04179afda01407cf399d7 2023-03-27 04:57:09,371 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/C of c239f1ca264c8e6332ec629bacfc12b1 into d3fb6c005fc04179afda01407cf399d7(size=11.8 K), total size for store is 23.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:09,371 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:09,371 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/C, priority=13, startTime=1679893028501; duration=0sec 2023-03-27 04:57:09,371 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:09,371 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:09,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:09,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:09,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:09,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:09,374 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:09,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:09,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:09,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:09,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,375 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:09,375 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:09,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:09,375 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,375 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:09,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:09,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:09,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:09,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:09,376 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:09,376 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:09,376 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:09,376 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:09,376 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:09,376 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,376 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:09,377 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:09,377 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,377 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:09,377 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:09,377 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,377 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:09,377 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:09,377 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,377 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,378 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,378 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:09,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78996040[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:09,379 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:09,379 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,379 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:09,379 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:09,379 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:09,380 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,380 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:09,380 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,380 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,380 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:09,380 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:09,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:09,382 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:09,382 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:09,382 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:09,382 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:09,382 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:09,383 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,383 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:09,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,383 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:09,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:09,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,384 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,384 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,386 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,386 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,387 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,387 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:09,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:09,387 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,387 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:09,387 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:09,387 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,387 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:09,387 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:09,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:09,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:09,387 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,387 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:09,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:09,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:09,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893089414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893089415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893089415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893089416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893089416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,487 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:09,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:09,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:09,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:09,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:09,490 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:09,490 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:09,490 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:09,490 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:09,490 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,490 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,491 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:09,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:09,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:09,491 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,492 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,492 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:09,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:09,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:09,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:09,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:09,492 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:09,492 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:09,492 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:09,492 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:09,492 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:09,492 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,492 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:09,493 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:09,493 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,493 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:09,493 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:09,493 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,493 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:09,493 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:09,493 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,493 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,494 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,494 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:09,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44fdc0c6[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:09,496 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:09,496 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,497 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,497 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:09,497 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:09,497 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:09,497 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,497 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:09,498 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:09,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:09,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:09,500 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:09,500 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:09,500 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:09,500 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:09,500 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:09,501 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,501 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,501 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:09,501 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,502 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,503 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,503 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:09,503 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,503 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:09,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:09,504 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:09,503 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:09,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:09,504 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,504 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,504 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,504 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:09,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:09,504 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:09,504 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:09,507 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/8d89fbcc896a404abaea45eb3a193209 2023-03-27 04:57:09,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/ac1a9458cc614813b35160f21ccee3e9 2023-03-27 04:57:09,604 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:09,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:09,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:09,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:09,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:09,607 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:09,607 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:09,607 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:09,607 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:09,607 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,607 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,608 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:09,608 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:09,608 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,608 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:09,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:09,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:09,609 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:09,609 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:09,609 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:09,609 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:09,609 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:09,609 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,609 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:09,610 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:09,610 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,610 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:09,610 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:09,610 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,610 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:09,610 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:09,611 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,611 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,611 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,611 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:09,612 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:09,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7468e6d8[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:09,612 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,613 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,613 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:09,613 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:09,613 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:09,613 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,613 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:09,614 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,614 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:09,614 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:09,614 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:09,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:09,616 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:09,616 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:09,616 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:09,616 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:09,616 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:09,617 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,617 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:09,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,617 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:09,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,618 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,618 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:09,620 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:09,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:09,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:09,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:09,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:09,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:09,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,620 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:09,620 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:09,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:09,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:09,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893089716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893089718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893089718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,721 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:09,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:09,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893089719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:09,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:09,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893089720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:09,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:09,723 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:09,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:09,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:09,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:09,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:09,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:09,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:09,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:09,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:09,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:09,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:09,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:09,725 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:09,725 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:09,725 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:09,725 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:09,725 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:09,725 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,726 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:09,726 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:09,726 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,726 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:09,726 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:09,726 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,726 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:09,726 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:09,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,727 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:09,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:09,728 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:09,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d0775b7[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:09,728 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,729 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,729 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:09,729 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,729 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:09,729 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:09,729 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:09,730 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:09,730 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,730 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,730 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:09,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:09,732 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:09,732 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:09,732 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:09,732 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:09,732 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:09,733 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,733 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,733 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:09,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,733 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,734 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,736 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,736 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,736 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:09,736 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:09,736 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:09,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:09,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:09,736 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,736 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:09,737 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:09,737 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:09,737 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,737 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:09,737 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,837 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:09,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:09,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:09,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:09,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:09,840 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:09,840 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:09,840 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:09,840 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:09,840 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,840 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,841 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:09,841 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:09,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:09,841 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,841 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:09,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:09,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:09,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:09,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:09,843 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:09,843 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:09,843 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:09,843 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:09,843 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:09,843 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,843 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:09,843 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:09,861 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,861 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:09,861 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:09,861 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,861 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:09,862 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:09,866 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,866 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,867 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,867 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:09,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e2f9537[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:09,868 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:09,868 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,869 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,869 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,869 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:09,869 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:09,869 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:09,869 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:09,870 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:09,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:09,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:09,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:09,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:09,872 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:09,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:09,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:09,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:09,872 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:09,874 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,874 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:09,874 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:09,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:09,874 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:09,875 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:09,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:09,877 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,877 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,877 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,877 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:09,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:09,877 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:09,877 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:09,877 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:09,877 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,877 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:09,877 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,877 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:09,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:09,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:09,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:09,877 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:09,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:09,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:09,925 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/9ac49fa75d6f41b9b00e416caf800146 2023-03-27 04:57:09,930 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/8d89fbcc896a404abaea45eb3a193209 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8d89fbcc896a404abaea45eb3a193209 2023-03-27 04:57:09,935 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8d89fbcc896a404abaea45eb3a193209, entries=150, sequenceid=134, filesize=11.7 K 2023-03-27 04:57:09,935 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/ac1a9458cc614813b35160f21ccee3e9 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/ac1a9458cc614813b35160f21ccee3e9 2023-03-27 04:57:09,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/ac1a9458cc614813b35160f21ccee3e9, entries=150, sequenceid=134, filesize=11.7 K 2023-03-27 04:57:09,941 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/9ac49fa75d6f41b9b00e416caf800146 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9ac49fa75d6f41b9b00e416caf800146 2023-03-27 04:57:09,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9ac49fa75d6f41b9b00e416caf800146, entries=150, sequenceid=134, filesize=11.7 K 2023-03-27 04:57:09,949 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for c239f1ca264c8e6332ec629bacfc12b1 in 849ms, sequenceid=134, compaction requested=true 2023-03-27 04:57:09,949 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:09,949 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:09,949 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:09,949 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:09,949 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:09,949 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:09,950 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:09,950 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:09,950 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/A is initiating minor compaction (all files) 2023-03-27 04:57:09,950 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/B is initiating minor compaction (all files) 2023-03-27 04:57:09,950 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/A in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,950 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/B in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,950 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ddc10a0be8604f31abe8df7361fc018b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/5833f1846ebd4f0aacae66e6772b796d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8d89fbcc896a404abaea45eb3a193209] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=35.1 K 2023-03-27 04:57:09,950 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/4d2f403efa67460980f892e181148204, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/38d3cc61e8eb4aab915c9aeefa597ed2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/ac1a9458cc614813b35160f21ccee3e9] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=35.1 K 2023-03-27 04:57:09,950 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting ddc10a0be8604f31abe8df7361fc018b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679893026535 2023-03-27 04:57:09,950 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 4d2f403efa67460980f892e181148204, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679893026535 2023-03-27 04:57:09,951 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 5833f1846ebd4f0aacae66e6772b796d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1679893027666 2023-03-27 04:57:09,951 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 38d3cc61e8eb4aab915c9aeefa597ed2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1679893027666 2023-03-27 04:57:09,951 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 8d89fbcc896a404abaea45eb3a193209, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679893028792 2023-03-27 04:57:09,951 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting ac1a9458cc614813b35160f21ccee3e9, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679893028792 2023-03-27 04:57:09,961 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#A#compaction#232 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:09,961 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#B#compaction#233 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:09,977 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:09,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:09,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:09,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:09,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:09,980 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:09,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:09,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:09,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:09,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:09,981 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,981 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:09,981 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:09,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:09,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:09,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:09,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:09,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:09,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:09,982 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:09,982 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:09,982 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:09,982 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:09,982 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:09,982 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:09,982 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:09,982 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-27 04:57:09,983 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:09,983 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:09,983 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:09,983 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:09,983 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:09,983 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:10,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:10,221 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:10,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893090232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893090233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893090233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893090233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893090234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893090335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893090337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893090337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893090337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893090338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,383 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/5ee666f666a04f67a370a40b7c54ebff as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/5ee666f666a04f67a370a40b7c54ebff 2023-03-27 04:57:10,387 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/B of c239f1ca264c8e6332ec629bacfc12b1 into 5ee666f666a04f67a370a40b7c54ebff(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:10,387 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:10,387 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/B, priority=13, startTime=1679893029949; duration=0sec 2023-03-27 04:57:10,387 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:10,387 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:10,388 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:10,388 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/C is initiating minor compaction (all files) 2023-03-27 04:57:10,388 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/C in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:10,388 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/d3fb6c005fc04179afda01407cf399d7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/4b29091dd1dd45c69f80d506ee97ce66, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9ac49fa75d6f41b9b00e416caf800146] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=35.1 K 2023-03-27 04:57:10,389 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting d3fb6c005fc04179afda01407cf399d7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1679893026535 2023-03-27 04:57:10,389 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 4b29091dd1dd45c69f80d506ee97ce66, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1679893027666 2023-03-27 04:57:10,389 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 9ac49fa75d6f41b9b00e416caf800146, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679893028792 2023-03-27 04:57:10,391 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/56555dfa88194654ba7fce1a2c8f9f57 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/56555dfa88194654ba7fce1a2c8f9f57 2023-03-27 04:57:10,391 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/3c2e88e8560e4675aefd7755e6257c08 2023-03-27 04:57:10,395 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#C#compaction#235 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:10,399 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/A of c239f1ca264c8e6332ec629bacfc12b1 into 56555dfa88194654ba7fce1a2c8f9f57(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:10,399 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:10,399 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/A, priority=13, startTime=1679893029949; duration=0sec 2023-03-27 04:57:10,399 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:10,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893090539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893090540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893090540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893090540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893090540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,807 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/9638a68da76043b39fed2f0fcb245242 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9638a68da76043b39fed2f0fcb245242 2023-03-27 04:57:10,811 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/C of c239f1ca264c8e6332ec629bacfc12b1 into 9638a68da76043b39fed2f0fcb245242(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:10,811 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:10,811 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/C, priority=13, startTime=1679893029949; duration=0sec 2023-03-27 04:57:10,811 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:10,816 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/90ac56cb61b6413ebae4cab4833f53f5 2023-03-27 04:57:10,830 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/d60035fa37ad4232b9bf5de2002f48f0 2023-03-27 04:57:10,835 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/3c2e88e8560e4675aefd7755e6257c08 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/3c2e88e8560e4675aefd7755e6257c08 2023-03-27 04:57:10,838 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/3c2e88e8560e4675aefd7755e6257c08, entries=150, sequenceid=150, filesize=11.8 K 2023-03-27 04:57:10,839 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/90ac56cb61b6413ebae4cab4833f53f5 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/90ac56cb61b6413ebae4cab4833f53f5 2023-03-27 04:57:10,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893090841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,843 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/90ac56cb61b6413ebae4cab4833f53f5, entries=150, sequenceid=150, filesize=11.8 K 2023-03-27 04:57:10,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,844 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/d60035fa37ad4232b9bf5de2002f48f0 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/d60035fa37ad4232b9bf5de2002f48f0 2023-03-27 04:57:10,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893090842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893090844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893090844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:10,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893090845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,848 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/d60035fa37ad4232b9bf5de2002f48f0, entries=150, sequenceid=150, filesize=11.8 K 2023-03-27 04:57:10,849 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for c239f1ca264c8e6332ec629bacfc12b1 in 867ms, sequenceid=150, compaction requested=false 2023-03-27 04:57:10,849 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:10,849 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:10,849 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:10,849 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:10,849 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,849 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:10,849 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:10,850 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,850 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:10,850 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:10,850 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:10,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:10,852 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:10,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44fe93c4[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:10,852 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:10,852 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:10,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:10,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:10,853 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:10,853 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:10,853 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:10,853 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,853 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:10,853 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:10,853 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:10,853 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:10,853 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:10,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:10,856 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:10,856 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:10,856 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:10,856 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:10,856 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:10,857 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:10,857 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:10,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:10,857 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:10,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:10,857 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:10,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:10,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:10,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,860 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:10,860 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:10,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:10,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:10,861 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:10,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:10,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:10,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:10,861 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:10,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:10,861 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:10,861 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:10,861 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:10,861 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,861 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:10,861 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:10,861 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:10,961 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:10,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:10,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:10,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:10,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:10,964 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:10,964 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:10,964 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:10,964 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:10,964 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:10,964 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:10,965 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,965 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:10,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:10,965 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:10,965 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:10,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:10,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:10,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:10,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:10,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:10,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:10,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:10,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:10,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:10,966 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:10,966 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:10,966 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-27 04:57:10,966 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:10,966 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:10,966 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:10,967 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:10,967 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:10,967 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:11,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:11,348 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:11,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893091355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893091356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893091357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893091358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893091358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,373 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/e99886582c2746a0a68c8b7682bba8c8 2023-03-27 04:57:11,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893091459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893091459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893091460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893091460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893091461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893091663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893091663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893091663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893091663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893091664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,781 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/bb62d6711ab7497cb64a7e28cd83045f 2023-03-27 04:57:11,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893091965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893091966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893091968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893091968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:11,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:11,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893091969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,190 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/c22a903c4f714a599c7ee73e4f27a53c 2023-03-27 04:57:12,194 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/e99886582c2746a0a68c8b7682bba8c8 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/e99886582c2746a0a68c8b7682bba8c8 2023-03-27 04:57:12,198 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/e99886582c2746a0a68c8b7682bba8c8, entries=150, sequenceid=170, filesize=11.8 K 2023-03-27 04:57:12,198 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/bb62d6711ab7497cb64a7e28cd83045f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/bb62d6711ab7497cb64a7e28cd83045f 2023-03-27 04:57:12,201 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/bb62d6711ab7497cb64a7e28cd83045f, entries=150, sequenceid=170, filesize=11.8 K 2023-03-27 04:57:12,202 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/c22a903c4f714a599c7ee73e4f27a53c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/c22a903c4f714a599c7ee73e4f27a53c 2023-03-27 04:57:12,207 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/c22a903c4f714a599c7ee73e4f27a53c, entries=150, sequenceid=170, filesize=11.8 K 2023-03-27 04:57:12,207 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for c239f1ca264c8e6332ec629bacfc12b1 in 1241ms, sequenceid=170, compaction requested=true 2023-03-27 04:57:12,208 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:12,208 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:12,208 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:12,208 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:12,208 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,208 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:12,208 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:12,209 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,209 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:12,209 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:12,209 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:12,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:12,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7793e3a5[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:12,210 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:12,210 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:12,211 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:12,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:12,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:12,211 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:12,211 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:12,211 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:12,211 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,211 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:12,211 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:12,211 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,211 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:12,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:12,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:12,211 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:12,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:12,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:12,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:12,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:12,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:12,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:12,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:12,213 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:12,213 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:12,213 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:12,213 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:12,213 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:12,222 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:12,222 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:12,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:12,223 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:12,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:12,223 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:12,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:12,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:12,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,224 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:12,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:12,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:12,228 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:12,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:12,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:12,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:12,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:12,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:12,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:12,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:12,228 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:12,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:12,228 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:12,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:12,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:12,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:12,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:12,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:12,328 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:12,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:12,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:12,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:12,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:12,331 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:12,331 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:12,331 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:12,331 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:12,332 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:12,332 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:12,333 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,333 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:12,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:12,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:12,333 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,333 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:12,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:12,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:12,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:12,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:12,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:12,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:12,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:12,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:12,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:12,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:12,335 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:12,335 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:12,335 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-27 04:57:12,335 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:12,335 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:12,335 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:12,335 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:12,335 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:12,335 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:12,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:12,471 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:12,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:12,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893092479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:12,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893092480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:12,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893092481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:12,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893092481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:12,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893092482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:12,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893092583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:12,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:12,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893092584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893092585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:12,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893092585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:12,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893092587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,754 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/82650da2334046ef974fc6510e6a4e7a 2023-03-27 04:57:12,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:12,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893092788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:12,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893092788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:12,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893092789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:12,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893092789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:12,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:12,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893092790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:13,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893093092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:13,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893093093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:13,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893093094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:13,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893093095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:13,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893093095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,168 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/25d39513b9c6424fa7923177cecf8fce 2023-03-27 04:57:13,578 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/bd0770e0956a4e96914d9fd607752bb7 2023-03-27 04:57:13,592 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/82650da2334046ef974fc6510e6a4e7a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/82650da2334046ef974fc6510e6a4e7a 2023-03-27 04:57:13,595 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/82650da2334046ef974fc6510e6a4e7a, entries=150, sequenceid=186, filesize=11.8 K 2023-03-27 04:57:13,596 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/25d39513b9c6424fa7923177cecf8fce as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/25d39513b9c6424fa7923177cecf8fce 2023-03-27 04:57:13,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:13,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893093598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:13,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893093598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,600 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:13,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893093599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,600 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:13,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893093599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,600 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:13,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893093599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,601 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/25d39513b9c6424fa7923177cecf8fce, entries=150, sequenceid=186, filesize=11.8 K 2023-03-27 04:57:13,601 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/bd0770e0956a4e96914d9fd607752bb7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/bd0770e0956a4e96914d9fd607752bb7 2023-03-27 04:57:13,604 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/bd0770e0956a4e96914d9fd607752bb7, entries=150, sequenceid=186, filesize=11.8 K 2023-03-27 04:57:13,605 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for c239f1ca264c8e6332ec629bacfc12b1 in 1270ms, sequenceid=186, compaction requested=true 2023-03-27 04:57:13,605 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:13,605 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:13,605 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:13,605 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:13,605 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,605 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:13,605 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:13,606 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:13,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:13,607 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:13,607 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:13,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:13,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:13,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:13,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11df8a2d[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:13,608 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:13,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:13,611 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:13,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:13,611 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:13,611 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:13,611 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:13,611 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,611 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:13,612 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,612 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:13,612 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:13,612 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:13,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:13,614 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:13,614 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:13,614 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:13,614 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:13,614 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:13,615 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:13,615 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:13,615 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:13,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:13,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:13,615 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:13,616 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,618 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,618 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:13,618 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:13,618 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,618 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:13,618 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:13,618 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:13,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:13,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:13,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:13,618 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:13,618 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:13,618 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:13,618 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:13,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:13,618 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:13,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:13,719 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:13,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:13,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:13,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:13,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:13,722 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:13,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:13,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:13,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:13,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:13,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:13,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,723 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:13,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:13,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:13,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:13,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:13,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:13,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:13,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:13,724 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:13,724 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:13,724 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:13,724 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:13,724 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:13,724 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:13,724 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:13,724 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-27 04:57:13,725 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:13,725 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:13,725 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:13,725 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:13,725 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:13,725 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:13,732 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/8f6520116ee7455eb54bd5716edd76b6 2023-03-27 04:57:14,141 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/ef69816aadb14c869d11d055ca5213a9 2023-03-27 04:57:14,150 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/30e9c317119b4121a1cd3e865ee84c51 2023-03-27 04:57:14,153 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/8f6520116ee7455eb54bd5716edd76b6 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8f6520116ee7455eb54bd5716edd76b6 2023-03-27 04:57:14,156 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8f6520116ee7455eb54bd5716edd76b6, entries=150, sequenceid=202, filesize=11.8 K 2023-03-27 04:57:14,157 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/ef69816aadb14c869d11d055ca5213a9 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/ef69816aadb14c869d11d055ca5213a9 2023-03-27 04:57:14,160 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/ef69816aadb14c869d11d055ca5213a9, entries=150, sequenceid=202, filesize=11.8 K 2023-03-27 04:57:14,160 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/30e9c317119b4121a1cd3e865ee84c51 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/30e9c317119b4121a1cd3e865ee84c51 2023-03-27 04:57:14,164 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/30e9c317119b4121a1cd3e865ee84c51, entries=150, sequenceid=202, filesize=11.8 K 2023-03-27 04:57:14,165 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for c239f1ca264c8e6332ec629bacfc12b1 in 441ms, sequenceid=202, compaction requested=true 2023-03-27 04:57:14,165 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:14,165 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:14,165 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:14,165 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:14,165 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,165 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:14,165 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:14,166 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,166 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:14,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:14,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34b330c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:14,168 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:14,168 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,168 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,169 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:14,169 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:14,169 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,169 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:14,169 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:14,169 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:14,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:14,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:14,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:14,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:14,172 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:14,172 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:14,172 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:14,172 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:14,172 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:14,173 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,173 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,173 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:14,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,174 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,175 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:14,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:14,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:14,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:14,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:14,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:14,177 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:14,177 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,177 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:14,177 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:14,177 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:14,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:14,277 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:14,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:14,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:14,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:14,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:14,280 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:14,280 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:14,280 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:14,280 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:14,280 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,280 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,281 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:14,281 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:14,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:14,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,281 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,281 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:14,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:14,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:14,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:14,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:14,282 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:14,282 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:14,282 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:14,283 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:14,283 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:14,283 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:14,283 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:14,283 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:14,283 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:14,283 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:14,283 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:14,283 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,284 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:14,284 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:14,284 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,284 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,285 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,285 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:14,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@339bf012[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:14,286 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:14,286 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,286 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,286 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:14,286 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:14,286 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:14,286 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,286 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:14,287 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,287 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,287 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:14,287 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:14,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:14,289 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:14,289 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:14,289 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:14,289 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:14,289 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:14,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:14,290 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,291 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,293 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,293 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,293 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,293 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,293 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:14,293 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:14,293 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:14,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:14,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:14,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:14,293 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,293 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:14,293 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,293 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:14,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:14,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:14,293 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:14,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:14,393 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:14,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:14,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:14,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:14,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:14,396 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:14,396 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:14,396 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:14,397 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:14,397 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,397 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,397 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:14,398 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:14,398 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,398 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:14,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:14,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:14,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:14,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:14,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:14,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:14,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:14,399 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:14,399 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:14,399 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:14,399 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:14,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:14,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:14,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:14,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:14,401 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,401 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,401 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,401 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:14,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:14,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bfdb30f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:14,402 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:14,402 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,404 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,404 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,404 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:14,404 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:14,404 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:14,404 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:14,404 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:14,405 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:14,405 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,405 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:14,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:14,406 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:14,407 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:14,407 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:14,407 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:14,407 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:14,407 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,408 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,408 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:14,408 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,409 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,411 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,411 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,411 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,411 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:14,411 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,411 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:14,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:14,411 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:14,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:14,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:14,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:14,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:14,411 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,412 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,412 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,412 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:14,412 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:14,512 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:14,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:14,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:14,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:14,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:14,515 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:14,515 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:14,515 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:14,515 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:14,515 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,515 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,516 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,516 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:14,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:14,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:14,516 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,516 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:14,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:14,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:14,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:14,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:14,517 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:14,517 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:14,517 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:14,517 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:14,517 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:14,517 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:14,517 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:14,517 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:14,517 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:14,517 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:14,517 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:14,517 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,517 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:14,517 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:14,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,518 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,518 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,518 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:14,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@361ec4bf[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:14,519 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:14,519 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,520 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,520 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,520 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:14,520 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:14,520 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:14,520 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:14,521 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:14,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:14,521 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:14,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:14,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:14,522 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:14,522 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:14,522 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:14,523 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:14,523 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:14,523 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,523 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,523 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:14,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:14,524 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,525 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,526 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,526 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,526 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:14,526 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:14,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:14,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:14,526 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:14,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:14,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:14,526 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,526 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,526 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:14,526 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,526 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:14,526 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,527 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:14,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:14,610 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:57:14,611 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:14,611 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:14,611 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:14,611 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:14,611 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:14,611 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:14,627 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:14,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:14,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:14,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:14,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:14,630 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:14,630 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:14,630 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:14,630 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:14,630 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,630 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:14,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893094625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893034623, max timestamp=1679893034630Num uniques 859; 2023-03-27 04:57:14,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893034623, max timestamp=1679893034630Num uniques 859; 2023-03-27 04:57:14,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:57:14,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893034623, max timestamp=1679893034630Num uniques 859; 2023-03-27 04:57:14,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:57:14,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:57:14,632 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:14,632 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:14,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:14,633 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,633 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:14,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:14,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:14,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:14,635 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:14,635 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:14,635 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:14,635 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:14,635 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:14,635 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:14,635 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:14,635 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:14,635 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,635 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:14,635 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:14,636 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:14,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:14,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c175223[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:14,637 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:14,637 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,638 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,638 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:14,638 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:14,638 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:14,638 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,638 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:14,639 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,639 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,639 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:14,639 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:14,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:14,641 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:14,641 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:14,641 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:14,641 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:14,641 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:14,642 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,642 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,642 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:14,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,642 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,643 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,644 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:14,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:14,645 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:14,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:14,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:14,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:14,645 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:14,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:14,645 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:14,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:14,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893094647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:14,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893094648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:14,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893094649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:14,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893094652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:14,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893094733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,745 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:14,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:14,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:14,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:14,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:14,748 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:14,748 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:14,748 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:14,748 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:14,748 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,748 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,749 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:14,749 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:14,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:14,749 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,749 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:14,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:14,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:14,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:14,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:14,750 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:14,750 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:14,750 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:14,750 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:14,751 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:14,751 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:14,751 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:14,751 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:14,751 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:14,751 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:14,751 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:14,751 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,751 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:14,751 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:14,752 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,752 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,753 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,753 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:14,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:14,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e024d3[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:14,754 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:14,754 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,755 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:14,755 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:14,755 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:14,755 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,755 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:14,756 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:14,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:14,757 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:14,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:14,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893094755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:14,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893094756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:14,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893094756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:14,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:14,759 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:14,759 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:14,759 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:14,759 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:14,759 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:14,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:14,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893094757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,760 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,760 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:14,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,760 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:14,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:14,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,761 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,761 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,763 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,763 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,763 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:14,763 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:14,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:14,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:14,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:14,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:14,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:14,763 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:14,763 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:14,763 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,763 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:14,763 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,763 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,763 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,763 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:14,863 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:14,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:14,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:14,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:14,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:14,866 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:14,866 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:14,866 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:14,866 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:14,867 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,867 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,867 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:14,867 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:14,868 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,868 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:14,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:14,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:14,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:14,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:14,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:14,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:14,869 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:14,869 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:14,869 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:14,869 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:14,869 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:14,869 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:14,869 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:14,869 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:14,869 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,869 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:14,869 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:14,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,870 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,870 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:14,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:14,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36b39a0b[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:14,871 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:14,871 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,872 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,872 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:14,872 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:14,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:14,872 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:14,873 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,873 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,873 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:14,873 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:14,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:14,875 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:14,875 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:14,875 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:14,875 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:14,875 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:14,876 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,876 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,876 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,876 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,877 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,878 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,879 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:14,879 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,879 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:14,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:14,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:14,879 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,879 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:14,879 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,879 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:14,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:14,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:14,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:14,879 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,879 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,879 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:14,879 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:14,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:14,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893094935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:14,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893094958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:14,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893094959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:14,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893094959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:14,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893094961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,979 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:14,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:14,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:14,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:14,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:14,982 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:14,982 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:14,982 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:14,982 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:14,983 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,983 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,983 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:14,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:14,984 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,984 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,984 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:14,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:14,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:14,985 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:14,985 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:14,985 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:14,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:14,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:14,986 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:14,986 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:14,986 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:14,986 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:14,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:14,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:14,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:14,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:14,987 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,987 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,987 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,987 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:14,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76c88b5e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:14,988 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:14,988 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,989 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,989 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:14,989 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:14,989 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,989 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:14,989 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:14,989 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,990 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,990 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:14,990 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:14,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:14,991 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:14,991 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:14,991 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:14,991 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:14,992 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:14,992 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,992 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,992 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:14,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:14,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:14,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,995 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,995 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,995 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:14,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:14,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:14,995 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:14,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:14,995 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:14,995 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:14,996 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:14,996 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:14,995 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:14,996 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:14,996 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:14,996 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:14,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:15,016 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/386bd6a96b0541c68a48994f9eb06fd1 2023-03-27 04:57:15,024 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/bca1cdf6c02941809a8448ee9f30715a 2023-03-27 04:57:15,096 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:15,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:15,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:15,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:15,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:15,099 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:15,099 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:15,099 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:15,099 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:15,099 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,099 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,100 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,100 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:15,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:15,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:15,100 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,100 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:15,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:15,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:15,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:15,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:15,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:15,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:15,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:15,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:15,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:15,101 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:15,101 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:15,101 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:15,102 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:15,102 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:15,102 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:15,102 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,102 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:15,102 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:15,102 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,102 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:15,103 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,103 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:15,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:15,104 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:15,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@209859ee[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:15,104 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,104 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,104 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,105 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:15,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:15,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:15,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:15,105 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:15,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:15,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:15,105 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:15,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:15,107 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:15,107 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:15,107 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:15,107 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:15,107 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:15,109 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,109 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,109 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:15,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:15,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:15,109 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:15,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:15,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:15,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:15,110 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:15,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:15,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:15,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,112 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:15,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:15,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:15,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,112 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:15,112 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:15,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:15,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:15,212 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:15,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:15,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:15,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:15,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:15,215 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:15,215 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:15,215 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:15,215 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:15,216 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,216 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,216 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:15,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:15,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:15,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,217 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,217 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,217 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:15,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:15,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:15,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:15,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:15,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:15,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:15,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:15,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:15,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:15,218 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:15,218 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:15,218 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:15,218 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:15,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:15,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:15,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,219 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:15,219 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:15,219 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,219 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:15,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:15,220 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,220 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:15,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:15,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:15,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:15,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:15,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21df2792[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:15,221 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:15,221 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,221 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:15,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:15,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:15,222 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,222 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:15,222 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:15,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:15,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:15,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:15,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:15,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:15,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:15,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:15,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:15,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:15,224 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:15,224 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:15,224 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:15,224 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:15,224 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:15,225 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,225 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:15,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:15,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,225 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:15,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:15,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:15,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:15,226 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:15,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,227 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:15,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:15,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,228 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:15,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:15,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:15,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:15,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:15,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:15,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:15,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,229 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,229 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,229 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:15,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:15,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:15,229 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:15,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:15,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42592 deadline: 1679893095237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:15,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42562 deadline: 1679893095262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:15,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893095263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:15,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893095264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:15,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893095265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,329 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:15,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:15,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:15,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:15,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:15,331 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:15,331 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:15,331 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:15,332 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:15,332 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,332 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,332 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,332 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:15,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:15,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:15,333 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,333 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:15,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:15,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:15,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:15,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:15,333 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:15,333 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:15,333 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:15,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:15,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:15,334 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:15,334 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:15,334 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:15,334 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:15,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:15,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:15,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:15,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:15,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,335 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:15,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:15,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:15,336 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:15,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b969260[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:15,336 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,337 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,337 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:15,337 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:15,337 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:15,337 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,337 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:15,338 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,338 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:15,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:15,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:15,338 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:15,338 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:15,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:15,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:15,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:15,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:15,339 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:15,340 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:15,340 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:15,340 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:15,340 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:15,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:15,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:15,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,341 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,342 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:15,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:15,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,344 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,344 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,344 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,344 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,344 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:15,344 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:15,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:15,344 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:15,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:15,344 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:15,344 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:15,344 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:15,344 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:15,344 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:15,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:15,433 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/46b3c110088f46b3bf36edfa19b7cb10 2023-03-27 04:57:15,436 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/386bd6a96b0541c68a48994f9eb06fd1 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/386bd6a96b0541c68a48994f9eb06fd1 2023-03-27 04:57:15,440 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/386bd6a96b0541c68a48994f9eb06fd1, entries=150, sequenceid=216, filesize=11.8 K 2023-03-27 04:57:15,440 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/bca1cdf6c02941809a8448ee9f30715a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/bca1cdf6c02941809a8448ee9f30715a 2023-03-27 04:57:15,443 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/bca1cdf6c02941809a8448ee9f30715a, entries=150, sequenceid=216, filesize=11.8 K 2023-03-27 04:57:15,444 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/46b3c110088f46b3bf36edfa19b7cb10 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/46b3c110088f46b3bf36edfa19b7cb10 2023-03-27 04:57:15,445 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:15,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:15,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:15,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/46b3c110088f46b3bf36edfa19b7cb10, entries=150, sequenceid=216, filesize=11.8 K 2023-03-27 04:57:15,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:15,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:15,447 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:15,447 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:15,447 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:15,448 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:15,448 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,448 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,448 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:15,448 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:15,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:15,449 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,449 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:15,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:15,449 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for c239f1ca264c8e6332ec629bacfc12b1 in 839ms, sequenceid=216, compaction requested=true 2023-03-27 04:57:15,449 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:15,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:15,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:15,449 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-27 04:57:15,449 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:15,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:15,449 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-27 04:57:15,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:15,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:15,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:15,450 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:15,450 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:15,450 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:15,450 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:15,450 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:15,450 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:15,450 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:15,450 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-27 04:57:15,451 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:15,451 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:15,451 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:15,451 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:15,451 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:15,451 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72474 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-27 04:57:15,451 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:15,451 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/B is initiating minor compaction (all files) 2023-03-27 04:57:15,451 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/B in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:15,451 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/5ee666f666a04f67a370a40b7c54ebff, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/90ac56cb61b6413ebae4cab4833f53f5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/bb62d6711ab7497cb64a7e28cd83045f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/25d39513b9c6424fa7923177cecf8fce, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/ef69816aadb14c869d11d055ca5213a9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/bca1cdf6c02941809a8448ee9f30715a] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=70.8 K 2023-03-27 04:57:15,451 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72474 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-27 04:57:15,451 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/A is initiating minor compaction (all files) 2023-03-27 04:57:15,451 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/A in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:15,451 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/56555dfa88194654ba7fce1a2c8f9f57, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/3c2e88e8560e4675aefd7755e6257c08, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/e99886582c2746a0a68c8b7682bba8c8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/82650da2334046ef974fc6510e6a4e7a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8f6520116ee7455eb54bd5716edd76b6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/386bd6a96b0541c68a48994f9eb06fd1] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=70.8 K 2023-03-27 04:57:15,452 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 5ee666f666a04f67a370a40b7c54ebff, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679893028792 2023-03-27 04:57:15,452 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 56555dfa88194654ba7fce1a2c8f9f57, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679893028792 2023-03-27 04:57:15,452 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 90ac56cb61b6413ebae4cab4833f53f5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1679893029104 2023-03-27 04:57:15,452 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 3c2e88e8560e4675aefd7755e6257c08, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1679893029104 2023-03-27 04:57:15,452 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting bb62d6711ab7497cb64a7e28cd83045f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1679893030229 2023-03-27 04:57:15,452 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting e99886582c2746a0a68c8b7682bba8c8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1679893030229 2023-03-27 04:57:15,453 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 25d39513b9c6424fa7923177cecf8fce, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1679893031351 2023-03-27 04:57:15,453 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 82650da2334046ef974fc6510e6a4e7a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1679893031351 2023-03-27 04:57:15,453 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting ef69816aadb14c869d11d055ca5213a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679893032476 2023-03-27 04:57:15,453 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 8f6520116ee7455eb54bd5716edd76b6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679893032476 2023-03-27 04:57:15,453 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting bca1cdf6c02941809a8448ee9f30715a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1679893034608 2023-03-27 04:57:15,453 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 386bd6a96b0541c68a48994f9eb06fd1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1679893034608 2023-03-27 04:57:15,471 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#B#compaction#251 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:15,481 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#A#compaction#252 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:15,494 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/ad49fcdd8bbd46bda4b1b63628cbade1 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ad49fcdd8bbd46bda4b1b63628cbade1 2023-03-27 04:57:15,497 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/A of c239f1ca264c8e6332ec629bacfc12b1 into ad49fcdd8bbd46bda4b1b63628cbade1(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:15,497 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:15,497 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/A, priority=10, startTime=1679893035449; duration=0sec 2023-03-27 04:57:15,498 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:15,498 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-27 04:57:15,499 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72474 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-27 04:57:15,499 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/C is initiating minor compaction (all files) 2023-03-27 04:57:15,499 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/C in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:15,499 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9638a68da76043b39fed2f0fcb245242, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/d60035fa37ad4232b9bf5de2002f48f0, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/c22a903c4f714a599c7ee73e4f27a53c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/bd0770e0956a4e96914d9fd607752bb7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/30e9c317119b4121a1cd3e865ee84c51, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/46b3c110088f46b3bf36edfa19b7cb10] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=70.8 K 2023-03-27 04:57:15,499 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 9638a68da76043b39fed2f0fcb245242, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679893028792 2023-03-27 04:57:15,499 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting d60035fa37ad4232b9bf5de2002f48f0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1679893029104 2023-03-27 04:57:15,500 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting c22a903c4f714a599c7ee73e4f27a53c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1679893030229 2023-03-27 04:57:15,500 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting bd0770e0956a4e96914d9fd607752bb7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1679893031351 2023-03-27 04:57:15,500 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 30e9c317119b4121a1cd3e865ee84c51, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679893032476 2023-03-27 04:57:15,500 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 46b3c110088f46b3bf36edfa19b7cb10, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1679893034608 2023-03-27 04:57:15,508 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#C#compaction#253 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:15,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:15,747 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:15,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:15,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893095767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:15,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893095767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:15,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893095769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:15,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42562 deadline: 1679893095769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:15,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893095771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,858 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/76c264eb0b14472a88e7c75cd476706b 2023-03-27 04:57:15,867 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/d6ca8a9f0a134be2b87e140545d88e02 2023-03-27 04:57:15,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:15,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893095870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:15,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42562 deadline: 1679893095872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:15,894 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/116201ea78544a63bdad492a020fe72d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/116201ea78544a63bdad492a020fe72d 2023-03-27 04:57:15,898 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/B of c239f1ca264c8e6332ec629bacfc12b1 into 116201ea78544a63bdad492a020fe72d(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:15,898 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:15,898 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/B, priority=10, startTime=1679893035449; duration=0sec 2023-03-27 04:57:15,898 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:15,915 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/9115b7b467a44570ad80968861d3a9e2 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9115b7b467a44570ad80968861d3a9e2 2023-03-27 04:57:15,918 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/C of c239f1ca264c8e6332ec629bacfc12b1 into 9115b7b467a44570ad80968861d3a9e2(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:15,918 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:15,918 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/C, priority=10, startTime=1679893035449; duration=0sec 2023-03-27 04:57:15,918 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:16,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:16,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893096076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:16,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42562 deadline: 1679893096076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,280 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/e9298c70d8a5465f9f6d010330e86428 2023-03-27 04:57:16,283 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/76c264eb0b14472a88e7c75cd476706b as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/76c264eb0b14472a88e7c75cd476706b 2023-03-27 04:57:16,288 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/76c264eb0b14472a88e7c75cd476706b, entries=150, sequenceid=251, filesize=11.8 K 2023-03-27 04:57:16,289 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/d6ca8a9f0a134be2b87e140545d88e02 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/d6ca8a9f0a134be2b87e140545d88e02 2023-03-27 04:57:16,292 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/d6ca8a9f0a134be2b87e140545d88e02, entries=150, sequenceid=251, filesize=11.8 K 2023-03-27 04:57:16,293 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/e9298c70d8a5465f9f6d010330e86428 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/e9298c70d8a5465f9f6d010330e86428 2023-03-27 04:57:16,299 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/e9298c70d8a5465f9f6d010330e86428, entries=150, sequenceid=251, filesize=11.8 K 2023-03-27 04:57:16,306 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for c239f1ca264c8e6332ec629bacfc12b1 in 856ms, sequenceid=251, compaction requested=false 2023-03-27 04:57:16,306 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:16,306 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:16,306 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:16,306 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:16,306 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,307 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:16,307 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:16,308 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,308 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,308 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,308 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:16,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39b66a37[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:16,310 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:16,310 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,311 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:16,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:16,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:16,311 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,311 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:16,312 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,312 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:16,312 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:16,312 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:16,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:16,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:16,315 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:16,315 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:16,315 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:16,315 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:16,315 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:16,316 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,316 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:16,316 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:16,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:16,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,317 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,318 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,322 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:16,322 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:16,322 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:16,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:16,322 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,322 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,322 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,322 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,322 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,322 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:16,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:16,322 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:16,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:16,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:16,322 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:16,323 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:16,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:16,379 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-27 04:57:16,380 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:16,380 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:16,380 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:16,380 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:16,380 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:16,380 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:16,387 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/4de2b01144b843298165307a796a1418 2023-03-27 04:57:16,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:16,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42562 deadline: 1679893096404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:16,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893096405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,423 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:16,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:16,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:16,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:16,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:16,425 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:16,425 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:16,425 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:16,425 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:16,426 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,426 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,426 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,426 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:16,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:16,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:16,426 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,426 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:16,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:16,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:16,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:16,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:16,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:16,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:16,428 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:16,428 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:16,428 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:16,428 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:16,428 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:16,428 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:16,428 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:16,428 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:16,428 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,428 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:16,428 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:16,429 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:16,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b0313c6[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:16,431 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:16,431 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,431 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,431 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:16,431 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:16,431 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:16,432 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,432 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:16,432 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,432 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,432 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:16,433 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:16,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:16,435 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:16,435 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:16,435 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:16,435 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:16,435 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:16,436 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,436 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,436 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,436 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,437 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,440 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:16,440 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:16,440 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,440 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,440 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,440 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:16,440 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:16,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:16,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:16,440 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,440 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:16,440 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,440 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,440 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:16,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:16,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:16,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:16,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42562 deadline: 1679893096507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:16,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893096508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,541 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:16,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:16,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:16,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:16,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:16,544 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:16,544 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:16,544 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:16,544 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:16,544 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,544 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,545 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,545 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:16,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:16,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:16,545 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,545 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:16,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:16,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:16,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:16,546 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:16,546 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:16,546 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:16,546 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:16,546 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:16,546 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:16,546 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:16,546 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:16,546 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:16,546 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:16,546 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:16,546 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,547 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:16,547 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:16,547 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,547 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,547 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,548 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:16,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b85c83[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:16,549 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:16,549 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,549 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,549 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,549 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:16,549 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:16,549 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:16,549 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:16,550 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:16,550 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:16,550 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,550 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:16,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:16,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:16,552 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:16,552 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:16,552 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:16,552 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:16,552 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:16,553 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,553 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,553 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:16,553 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:16,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,554 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,556 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:16,556 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:16,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:16,556 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:16,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:16,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:16,556 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:16,556 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,556 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:16,556 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:16,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:16,556 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,556 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,556 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,556 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:16,556 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:16,657 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:16,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:16,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:16,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:16,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:16,659 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:16,659 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:16,659 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:16,659 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:16,660 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,660 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,660 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:16,660 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:16,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:16,661 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,661 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:16,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:16,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:16,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:16,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:16,662 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:16,662 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:16,662 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:16,662 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:16,662 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:16,662 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:16,662 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:16,662 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:16,662 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:16,662 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:16,662 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:16,662 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,662 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:16,662 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:16,663 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,663 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,664 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,664 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:16,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:16,664 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:16,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c82db06[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:16,664 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,665 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:16,665 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:16,665 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:16,665 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,665 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:16,666 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:16,666 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:16,666 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:16,666 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:16,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:16,668 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:16,668 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:16,669 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:16,669 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:16,669 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:16,670 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,670 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,670 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:16,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:16,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:16,670 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,672 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,674 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:16,674 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:16,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:16,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:16,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:16,674 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:16,674 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:16,674 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:16,674 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:16,674 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:16,674 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,674 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:16,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:16,675 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,675 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,675 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:16,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42562 deadline: 1679893096710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:16,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893096711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:16,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893096771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,775 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:16,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:16,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:16,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893096775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:16,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893096776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:16,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:16,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:16,778 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:16,778 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:16,778 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:16,778 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:16,778 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,778 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,780 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:16,780 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:16,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:16,780 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,780 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:16,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:16,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:16,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:16,781 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:16,781 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:16,781 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:16,781 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:16,781 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:16,781 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:16,781 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:16,781 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:16,781 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:16,782 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:16,782 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:16,782 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,782 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:16,782 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:16,782 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,783 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,783 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:16,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:16,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36eff973[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:16,784 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:16,784 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,785 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,785 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:16,785 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:16,785 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:16,785 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,785 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:16,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,786 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:16,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,786 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:16,786 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:16,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:16,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:16,788 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:16,788 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:16,788 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:16,788 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:16,788 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:16,789 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,789 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,789 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:16,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:16,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:16,789 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,790 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,791 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,791 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,791 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,791 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,791 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:16,791 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:16,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:16,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:16,791 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:16,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:16,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:16,791 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:16,792 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,792 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,792 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:16,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:16,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:16,792 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:16,798 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/b3983c9980cb4c14914f908a2f323f3d 2023-03-27 04:57:16,892 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:16,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:16,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:16,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:16,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:16,896 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:16,896 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:16,896 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:16,896 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:16,896 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,896 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,897 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,897 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,897 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:16,897 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:16,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:16,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:16,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:16,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:16,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:16,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:16,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:16,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:16,899 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:16,899 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:16,899 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:16,899 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:16,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:16,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:16,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:16,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:16,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,900 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,901 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,901 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:16,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75b1414b[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:16,902 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:16,902 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,903 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,903 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:16,903 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:16,903 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,903 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:16,903 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:16,904 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:16,904 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,904 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,904 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:16,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:16,906 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:16,906 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:16,906 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:16,906 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:16,906 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:16,907 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,907 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:16,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,907 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:16,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:16,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:16,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:16,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:16,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,908 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,909 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:16,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,911 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,911 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,911 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:16,911 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:16,911 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,911 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:16,911 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:16,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:16,911 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:16,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:16,911 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:16,911 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:16,911 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:16,911 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:16,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:16,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:16,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:16,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:17,012 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:17,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:17,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:17,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:17,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42562 deadline: 1679893097014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:17,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:17,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893097015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:17,016 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:17,016 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:17,016 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:17,016 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:17,016 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,016 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,017 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:17,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:17,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:17,017 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,017 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:17,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:17,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:17,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:17,018 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:17,018 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:17,018 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:17,018 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:17,018 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:17,018 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:17,018 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:17,018 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:17,018 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:17,018 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,019 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,020 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,020 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:17,021 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:17,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34ac473a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:17,021 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,021 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,022 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:17,022 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,022 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:17,022 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:17,022 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:17,022 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,022 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,023 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:17,023 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:17,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:17,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:17,025 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:17,025 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:17,025 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:17,025 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:17,025 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:17,026 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,026 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,026 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:17,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:17,026 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,028 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,029 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,029 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,029 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:17,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:17,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:17,029 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,029 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:17,029 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:17,029 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,029 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:17,029 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:17,029 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:17,029 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,029 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:17,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:17,130 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:17,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:17,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:17,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:17,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:17,132 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:17,132 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:17,132 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:17,132 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:17,132 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,133 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,134 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,134 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:17,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:17,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:17,134 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,134 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:17,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:17,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:17,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:17,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:17,135 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:17,135 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:17,135 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:17,135 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:17,135 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:17,135 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:17,135 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:17,136 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:17,136 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:17,136 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:17,136 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:17,136 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,136 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:17,136 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:17,138 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,138 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,138 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,138 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:17,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e5ce042[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:17,139 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:17,139 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,140 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,140 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:17,140 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:17,140 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:17,140 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,140 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:17,140 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:17,141 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,141 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,141 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:17,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:17,143 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:17,143 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:17,143 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:17,143 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:17,143 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:17,144 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,144 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,144 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:17,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:17,144 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,145 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,147 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:17,147 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:17,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:17,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:17,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:17,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:17,147 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:17,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:17,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:17,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:17,147 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,147 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:17,147 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,147 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:17,147 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,147 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,147 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,148 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,148 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,219 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/deef1b23cba8495a839e0a24181baeb2 2023-03-27 04:57:17,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/4de2b01144b843298165307a796a1418 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/4de2b01144b843298165307a796a1418 2023-03-27 04:57:17,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/4de2b01144b843298165307a796a1418, entries=200, sequenceid=267, filesize=14.3 K 2023-03-27 04:57:17,226 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/b3983c9980cb4c14914f908a2f323f3d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/b3983c9980cb4c14914f908a2f323f3d 2023-03-27 04:57:17,230 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/b3983c9980cb4c14914f908a2f323f3d, entries=150, sequenceid=267, filesize=11.9 K 2023-03-27 04:57:17,231 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/deef1b23cba8495a839e0a24181baeb2 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/deef1b23cba8495a839e0a24181baeb2 2023-03-27 04:57:17,236 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/deef1b23cba8495a839e0a24181baeb2, entries=150, sequenceid=267, filesize=11.9 K 2023-03-27 04:57:17,237 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for c239f1ca264c8e6332ec629bacfc12b1 in 857ms, sequenceid=267, compaction requested=true 2023-03-27 04:57:17,237 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:17,237 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:17,237 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:17,237 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:17,237 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:17,237 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:17,238 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39210 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:17,238 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/A is initiating minor compaction (all files) 2023-03-27 04:57:17,238 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/A in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:17,238 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ad49fcdd8bbd46bda4b1b63628cbade1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/76c264eb0b14472a88e7c75cd476706b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/4de2b01144b843298165307a796a1418] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=38.3 K 2023-03-27 04:57:17,238 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36770 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:17,238 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/B is initiating minor compaction (all files) 2023-03-27 04:57:17,238 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting ad49fcdd8bbd46bda4b1b63628cbade1, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1679893034608 2023-03-27 04:57:17,238 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/B in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:17,238 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/116201ea78544a63bdad492a020fe72d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/d6ca8a9f0a134be2b87e140545d88e02, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/b3983c9980cb4c14914f908a2f323f3d] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=35.9 K 2023-03-27 04:57:17,239 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 76c264eb0b14472a88e7c75cd476706b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679893034647 2023-03-27 04:57:17,239 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 116201ea78544a63bdad492a020fe72d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1679893034608 2023-03-27 04:57:17,239 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 4de2b01144b843298165307a796a1418, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679893035766 2023-03-27 04:57:17,239 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting d6ca8a9f0a134be2b87e140545d88e02, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679893034647 2023-03-27 04:57:17,239 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting b3983c9980cb4c14914f908a2f323f3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679893035766 2023-03-27 04:57:17,246 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#B#compaction#259 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:17,246 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#A#compaction#260 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:17,247 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:17,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:17,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:17,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:17,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:17,274 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:17,274 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:17,274 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:17,274 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:17,274 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,274 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,275 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,275 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:17,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:17,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:17,275 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,276 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:17,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:17,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:17,311 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/be44783c9cc94c7295b833f78ff1be6e as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/be44783c9cc94c7295b833f78ff1be6e 2023-03-27 04:57:17,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:17,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:17,313 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:17,313 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:17,313 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:17,317 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/B of c239f1ca264c8e6332ec629bacfc12b1 into be44783c9cc94c7295b833f78ff1be6e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:17,317 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:17,317 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/B, priority=13, startTime=1679893037237; duration=0sec 2023-03-27 04:57:17,318 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:17,318 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:17,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:17,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:17,319 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:17,319 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:17,319 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-27 04:57:17,319 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:17,319 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36770 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:17,319 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/C is initiating minor compaction (all files) 2023-03-27 04:57:17,320 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/C in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:17,320 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9115b7b467a44570ad80968861d3a9e2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/e9298c70d8a5465f9f6d010330e86428, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/deef1b23cba8495a839e0a24181baeb2] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=35.9 K 2023-03-27 04:57:17,319 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:17,320 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 9115b7b467a44570ad80968861d3a9e2, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1679893034608 2023-03-27 04:57:17,320 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:17,320 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:17,320 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:17,320 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:17,320 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting e9298c70d8a5465f9f6d010330e86428, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679893034647 2023-03-27 04:57:17,321 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting deef1b23cba8495a839e0a24181baeb2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679893035766 2023-03-27 04:57:17,326 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#C#compaction#262 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:17,339 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/638391d2e3dd4ba98ca8592c04205492 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/638391d2e3dd4ba98ca8592c04205492 2023-03-27 04:57:17,344 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/A of c239f1ca264c8e6332ec629bacfc12b1 into 638391d2e3dd4ba98ca8592c04205492(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:17,344 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:17,344 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/A, priority=13, startTime=1679893037237; duration=0sec 2023-03-27 04:57:17,344 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:17,371 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/16a6959672094ed69bda02fa50a484de 2023-03-27 04:57:17,392 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/b47e532304584ef8bd25be819d4af10c 2023-03-27 04:57:17,402 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/0b434c51995b4c92a92798d30d45420a 2023-03-27 04:57:17,406 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/16a6959672094ed69bda02fa50a484de as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/16a6959672094ed69bda02fa50a484de 2023-03-27 04:57:17,410 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/16a6959672094ed69bda02fa50a484de, entries=150, sequenceid=287, filesize=11.9 K 2023-03-27 04:57:17,411 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/b47e532304584ef8bd25be819d4af10c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/b47e532304584ef8bd25be819d4af10c 2023-03-27 04:57:17,417 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/b47e532304584ef8bd25be819d4af10c, entries=150, sequenceid=287, filesize=11.9 K 2023-03-27 04:57:17,418 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/0b434c51995b4c92a92798d30d45420a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/0b434c51995b4c92a92798d30d45420a 2023-03-27 04:57:17,426 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/0b434c51995b4c92a92798d30d45420a, entries=150, sequenceid=287, filesize=11.9 K 2023-03-27 04:57:17,427 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for c239f1ca264c8e6332ec629bacfc12b1 in 108ms, sequenceid=287, compaction requested=false 2023-03-27 04:57:17,427 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:17,427 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:17,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:17,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:17,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,430 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:17,430 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:17,431 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,431 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,432 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,432 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:17,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:17,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5731a69b[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:17,434 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:17,434 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,435 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,435 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:17,435 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:17,435 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:17,436 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,436 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:17,436 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:17,437 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,437 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:17,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,437 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:17,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:17,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:17,440 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:17,440 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:17,440 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:17,440 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:17,440 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:17,450 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,450 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:17,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,450 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,452 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,454 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,458 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:17,458 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:17,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:17,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:17,458 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:17,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:17,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:17,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:17,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,459 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:17,459 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:17,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:17,524 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:57:17,525 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:17,525 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:17,525 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:17,525 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:17,525 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:17,525 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:17,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:17,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893097553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893037547, max timestamp=1679893037555Num uniques 859; 2023-03-27 04:57:17,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893037547, max timestamp=1679893037555Num uniques 859; 2023-03-27 04:57:17,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:57:17,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:57:17,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893037547, max timestamp=1679893037555Num uniques 859; 2023-03-27 04:57:17,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:57:17,559 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:17,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:17,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:17,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:17,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:17,562 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:17,562 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:17,562 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:17,562 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:17,562 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,562 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,563 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:17,563 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:17,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:17,563 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,563 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:17,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:17,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:17,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:17,565 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:17,565 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:17,565 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:17,565 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:17,565 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:17,565 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:17,565 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:17,565 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:17,565 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:17,565 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:17,565 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:17,565 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,565 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:17,565 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:17,566 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,566 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,567 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,567 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:17,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:17,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28f50e2f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:17,568 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:17,568 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,569 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,569 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:17,569 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:17,569 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:17,570 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:17,570 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:17,570 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:17,578 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,579 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:17,579 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:17,579 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:17,579 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:17,579 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:17,579 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:17,579 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:17,583 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,583 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,583 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:17,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,584 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,585 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:17,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:17,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:17,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:17,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,587 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:17,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:17,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:17,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,588 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:17,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:17,588 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,588 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,588 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,588 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:17,588 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:17,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:17,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42562 deadline: 1679893097610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:17,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893097657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,688 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:17,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:17,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:17,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:17,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:17,691 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:17,691 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:17,691 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:17,691 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:17,691 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,691 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,692 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,692 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:17,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:17,692 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,692 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:17,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:17,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:17,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:17,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:17,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:17,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:17,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:17,694 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:17,694 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:17,694 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:17,694 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:17,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:17,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:17,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:17,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:17,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,696 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:17,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eb6099c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:17,697 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:17,697 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,698 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,698 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:17,698 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:17,698 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:17,698 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,698 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:17,700 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:17,700 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:17,700 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:17,700 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:17,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:17,702 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:17,703 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:17,703 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:17,703 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:17,703 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:17,704 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,704 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,704 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:17,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:17,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:17,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,705 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,706 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,707 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:17,707 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:17,707 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:17,707 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:17,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:17,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:17,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:17,707 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,707 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,707 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,707 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,707 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:17,708 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,708 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:17,708 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:17,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:17,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42562 deadline: 1679893097714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,756 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/57771109568344f9b5aa7df32c1df13d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/57771109568344f9b5aa7df32c1df13d 2023-03-27 04:57:17,760 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/C of c239f1ca264c8e6332ec629bacfc12b1 into 57771109568344f9b5aa7df32c1df13d(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:17,760 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:17,760 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/C, priority=13, startTime=1679893037237; duration=0sec 2023-03-27 04:57:17,760 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:17,808 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:17,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:17,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:17,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:17,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:17,810 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:17,810 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:17,810 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:17,810 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:17,811 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,811 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,811 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:17,811 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:17,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,812 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,812 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:17,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:17,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:17,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:17,813 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:17,813 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:17,813 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:17,813 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:17,813 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:17,813 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:17,813 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:17,813 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:17,813 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:17,813 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:17,813 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:17,813 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,813 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:17,813 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:17,814 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,814 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,814 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,814 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:17,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@711b8f8[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:17,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:17,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,816 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,816 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:17,816 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:17,816 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:17,816 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,816 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:17,817 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,817 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:17,817 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,817 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:17,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:17,818 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:17,818 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:17,818 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:17,818 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:17,818 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:17,819 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,819 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,820 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:17,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,820 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,821 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,822 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,822 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,822 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,822 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:17,822 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:17,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:17,823 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,823 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,823 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,823 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:17,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:17,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:17,823 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:17,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:17,823 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:17,823 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:17,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:17,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893097861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:17,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42562 deadline: 1679893097918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,923 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:17,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:17,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:17,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:17,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:17,925 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:17,925 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:17,925 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:17,925 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:17,926 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,926 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,926 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:17,926 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:17,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:17,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,927 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,927 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:17,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:17,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:17,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:17,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:17,927 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:17,927 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:17,928 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:17,928 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:17,928 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:17,928 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:17,928 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:17,928 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:17,928 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:17,928 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:17,928 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:17,928 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,928 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:17,928 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:17,929 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,929 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,929 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,929 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:17,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dcec3ff[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:17,930 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:17,930 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,931 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,931 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,931 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:17,931 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:17,931 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:17,931 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:17,932 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,932 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:17,932 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:17,932 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:17,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:17,934 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:17,934 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:17,934 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:17,934 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:17,934 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:17,935 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,935 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,935 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:17,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,935 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:17,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:17,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:17,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,937 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,938 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:17,938 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:17,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:17,938 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:17,938 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,938 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,938 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:17,938 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:17,938 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,939 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:17,939 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:17,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:17,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:17,939 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:17,939 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:17,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:17,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:17,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:17,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/f27366f115494d02964f5c836731a940 2023-03-27 04:57:18,039 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:18,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:18,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:18,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:18,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:18,043 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:18,043 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:18,043 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:18,043 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:18,044 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,044 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,044 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:18,044 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:18,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:18,045 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,045 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:18,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:18,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:18,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:18,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:18,046 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:18,046 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:18,046 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:18,046 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:18,046 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:18,046 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:18,046 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:18,046 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:18,046 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:18,046 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:18,046 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:18,046 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,047 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:18,047 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:18,048 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,048 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:18,048 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,048 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:18,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:18,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dca2a1f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:18,049 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:18,049 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,050 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,050 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:18,050 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:18,050 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:18,050 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,050 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:18,051 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,051 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:18,051 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:18,051 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:18,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:18,053 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:18,053 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:18,053 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:18,053 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:18,053 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:18,054 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,054 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,054 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:18,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:18,055 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,056 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,057 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,057 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,057 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,057 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,057 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:18,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:18,057 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:18,057 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:18,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:18,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:18,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:18,057 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,057 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,057 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:18,058 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:18,058 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:18,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:18,158 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:18,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:18,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:18,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:18,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:18,160 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:18,160 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:18,160 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:18,160 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:18,161 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,161 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,161 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:18,161 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:18,161 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,161 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:18,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:18,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:18,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:18,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:18,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:18,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:18,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:18,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:18,162 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:18,163 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:18,163 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:18,163 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:18,163 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:18,163 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:18,163 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,163 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:18,163 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:18,164 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,164 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:18,164 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,164 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:18,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:18,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@137e32d3[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:18,165 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:18,165 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:18,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893098164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,166 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,166 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:18,166 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,166 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:18,166 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:18,166 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:18,167 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:18,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:18,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:18,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:18,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:18,168 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:18,168 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:18,168 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:18,169 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:18,169 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:18,169 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,169 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:18,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,169 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:18,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:18,170 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,171 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,172 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:18,172 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:18,172 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:18,172 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,172 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:18,172 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,173 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,173 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,173 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:18,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:18,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:18,173 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,173 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:18,173 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:18,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:18,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42562 deadline: 1679893098222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,273 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:18,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:18,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:18,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:18,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:18,276 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:18,276 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:18,276 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:18,276 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:18,276 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,276 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,277 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:18,277 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:18,277 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,277 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:18,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:18,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:18,278 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:18,278 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:18,278 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:18,278 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:18,278 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:18,278 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:18,278 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:18,278 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:18,278 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:18,278 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:18,278 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:18,279 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,279 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:18,279 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:18,279 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,280 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:18,280 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:18,280 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:18,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:18,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19229e3c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:18,281 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:18,281 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,281 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,282 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:18,282 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:18,282 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:18,282 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,282 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:18,282 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:18,282 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:18,282 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:18,282 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:18,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:18,285 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:18,286 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:18,286 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:18,286 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:18,286 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:18,286 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,287 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,287 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:18,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:18,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:18,287 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,288 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:18,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,290 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:18,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:18,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:18,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:18,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:18,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,290 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:18,290 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:18,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:18,290 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:18,349 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/d791984b08974d8686afb6a426dc5dee 2023-03-27 04:57:18,359 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/677ab0204903443593a6315cfd301a94 2023-03-27 04:57:18,365 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/f27366f115494d02964f5c836731a940 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/f27366f115494d02964f5c836731a940 2023-03-27 04:57:18,367 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/f27366f115494d02964f5c836731a940, entries=200, sequenceid=299, filesize=14.3 K 2023-03-27 04:57:18,368 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/d791984b08974d8686afb6a426dc5dee as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/d791984b08974d8686afb6a426dc5dee 2023-03-27 04:57:18,370 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/d791984b08974d8686afb6a426dc5dee, entries=150, sequenceid=299, filesize=11.9 K 2023-03-27 04:57:18,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/677ab0204903443593a6315cfd301a94 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/677ab0204903443593a6315cfd301a94 2023-03-27 04:57:18,373 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/677ab0204903443593a6315cfd301a94, entries=150, sequenceid=299, filesize=11.9 K 2023-03-27 04:57:18,374 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for c239f1ca264c8e6332ec629bacfc12b1 in 850ms, sequenceid=299, compaction requested=true 2023-03-27 04:57:18,374 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:18,374 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:18,374 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:18,374 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:18,374 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:18,375 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:18,375 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39612 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:18,375 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:18,375 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/A is initiating minor compaction (all files) 2023-03-27 04:57:18,375 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/B is initiating minor compaction (all files) 2023-03-27 04:57:18,375 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/A in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:18,375 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/B in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:18,375 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/638391d2e3dd4ba98ca8592c04205492, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/16a6959672094ed69bda02fa50a484de, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/f27366f115494d02964f5c836731a940] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=38.7 K 2023-03-27 04:57:18,375 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/be44783c9cc94c7295b833f78ff1be6e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/b47e532304584ef8bd25be819d4af10c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/d791984b08974d8686afb6a426dc5dee] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=36.3 K 2023-03-27 04:57:18,376 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 638391d2e3dd4ba98ca8592c04205492, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679893035766 2023-03-27 04:57:18,376 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting be44783c9cc94c7295b833f78ff1be6e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679893035766 2023-03-27 04:57:18,376 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 16a6959672094ed69bda02fa50a484de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1679893036402 2023-03-27 04:57:18,376 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting b47e532304584ef8bd25be819d4af10c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1679893036402 2023-03-27 04:57:18,376 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting f27366f115494d02964f5c836731a940, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1679893037519 2023-03-27 04:57:18,376 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting d791984b08974d8686afb6a426dc5dee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1679893037519 2023-03-27 04:57:18,382 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#B#compaction#269 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:18,382 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#A#compaction#268 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:18,390 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:18,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:18,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:18,393 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/4ea4715684ba4d639d89a207d3c010e5 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/4ea4715684ba4d639d89a207d3c010e5 2023-03-27 04:57:18,393 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/7c3f80e937604805b8fcd03545104433 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/7c3f80e937604805b8fcd03545104433 2023-03-27 04:57:18,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:18,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:18,394 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:18,394 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:18,394 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:18,394 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:18,396 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,396 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,397 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,397 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:18,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:18,397 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,397 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:18,400 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/B of c239f1ca264c8e6332ec629bacfc12b1 into 7c3f80e937604805b8fcd03545104433(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:18,400 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/A of c239f1ca264c8e6332ec629bacfc12b1 into 4ea4715684ba4d639d89a207d3c010e5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:18,400 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:18,400 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/B, priority=13, startTime=1679893038374; duration=0sec 2023-03-27 04:57:18,400 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:18,400 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/A, priority=13, startTime=1679893038374; duration=0sec 2023-03-27 04:57:18,400 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:18,400 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:18,400 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:18,401 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:18,401 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/C is initiating minor compaction (all files) 2023-03-27 04:57:18,401 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/C in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:18,401 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/57771109568344f9b5aa7df32c1df13d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/0b434c51995b4c92a92798d30d45420a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/677ab0204903443593a6315cfd301a94] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=36.3 K 2023-03-27 04:57:18,401 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 57771109568344f9b5aa7df32c1df13d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679893035766 2023-03-27 04:57:18,402 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 0b434c51995b4c92a92798d30d45420a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1679893036402 2023-03-27 04:57:18,402 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 677ab0204903443593a6315cfd301a94, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1679893037519 2023-03-27 04:57:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:18,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:18,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:18,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:18,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:18,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:18,403 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:18,406 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:18,406 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:18,406 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:18,406 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:18,406 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-27 04:57:18,406 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:18,406 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:18,407 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:18,407 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:18,407 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:18,407 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:18,409 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#C#compaction#270 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:18,423 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/b57acd8f9c11402bad2d401c26e2074a 2023-03-27 04:57:18,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:18,677 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:18,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:18,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893098696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:18,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42562 deadline: 1679893098725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:18,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42610 deadline: 1679893098776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,779 DEBUG [Thread-1660] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4131 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., hostname=jenkins-hbase20.apache.org,44923,1679892955505, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:57:18,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:18,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42590 deadline: 1679893098781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,783 DEBUG [Thread-1664] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., hostname=jenkins-hbase20.apache.org,44923,1679892955505, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:57:18,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:18,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42556 deadline: 1679893098795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,798 DEBUG [Thread-1658] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., hostname=jenkins-hbase20.apache.org,44923,1679892955505, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:57:18,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:18,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893098799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:18,824 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/9813b20303744ebf9340988b1c1cb293 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9813b20303744ebf9340988b1c1cb293 2023-03-27 04:57:18,827 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/C of c239f1ca264c8e6332ec629bacfc12b1 into 9813b20303744ebf9340988b1c1cb293(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:18,827 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:18,827 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/C, priority=13, startTime=1679893038375; duration=0sec 2023-03-27 04:57:18,827 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:18,838 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/f40e3039f90d4a8c990e6da0618dce5f 2023-03-27 04:57:19,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:19,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893099004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,247 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/1f59565948a34c81bbf18981fd3865b7 2023-03-27 04:57:19,250 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/b57acd8f9c11402bad2d401c26e2074a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/b57acd8f9c11402bad2d401c26e2074a 2023-03-27 04:57:19,253 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/b57acd8f9c11402bad2d401c26e2074a, entries=150, sequenceid=338, filesize=11.9 K 2023-03-27 04:57:19,254 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/f40e3039f90d4a8c990e6da0618dce5f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/f40e3039f90d4a8c990e6da0618dce5f 2023-03-27 04:57:19,257 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/f40e3039f90d4a8c990e6da0618dce5f, entries=150, sequenceid=338, filesize=11.9 K 2023-03-27 04:57:19,257 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/1f59565948a34c81bbf18981fd3865b7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/1f59565948a34c81bbf18981fd3865b7 2023-03-27 04:57:19,260 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/1f59565948a34c81bbf18981fd3865b7, entries=150, sequenceid=338, filesize=11.9 K 2023-03-27 04:57:19,260 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for c239f1ca264c8e6332ec629bacfc12b1 in 854ms, sequenceid=338, compaction requested=false 2023-03-27 04:57:19,261 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:19,261 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:19,261 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:19,261 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:19,261 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,261 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:19,261 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:19,262 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,262 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,262 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:19,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@325867dc[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:19,263 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:19,263 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,264 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:19,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:19,264 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,264 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:19,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:19,264 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:19,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,264 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:19,265 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:19,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:19,266 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:19,266 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:19,266 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:19,266 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:19,266 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:19,267 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,267 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,267 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,267 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,268 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,270 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,270 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:19,270 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:19,270 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:19,270 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:19,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:19,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:19,270 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,271 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,271 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,271 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,271 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:19,271 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:19,271 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:19,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:19,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:19,308 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-27 04:57:19,308 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:19,308 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:19,309 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:19,309 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:19,309 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:19,309 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:19,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:19,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893099348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,371 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:19,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:19,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:19,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:19,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:19,373 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:19,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:19,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:19,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:19,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,375 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:19,375 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:19,375 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,375 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:19,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:19,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:19,376 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:19,376 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:19,377 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:19,377 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:19,377 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:19,377 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:19,377 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:19,377 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:19,377 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:19,377 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:19,377 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:19,377 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,377 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:19,377 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:19,378 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,378 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,378 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,378 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:19,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ece295a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:19,379 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:19,379 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,380 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,380 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:19,380 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,380 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:19,380 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:19,380 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:19,381 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,381 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:19,381 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,381 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:19,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:19,383 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:19,383 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:19,383 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:19,383 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:19,383 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:19,383 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,383 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,383 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:19,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,384 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,385 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,393 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,393 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,393 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:19,393 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,393 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,393 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,393 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,394 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,393 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:19,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:19,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:19,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:19,394 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:19,394 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:19,394 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:19,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:19,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:19,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893099450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,494 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:19,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:19,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:19,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:19,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:19,496 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:19,496 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:19,497 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:19,497 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:19,499 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,499 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,500 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:19,500 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:19,504 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,504 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:19,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:19,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:19,505 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:19,505 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:19,505 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:19,505 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:19,505 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:19,505 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:19,505 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:19,506 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:19,506 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:19,506 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:19,506 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:19,506 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,506 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:19,506 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:19,510 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,510 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,510 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,510 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:19,511 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:19,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cc46cfa[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:19,511 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,512 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:19,512 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,512 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:19,512 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:19,512 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:19,512 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:19,512 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:19,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,512 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:19,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:19,514 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:19,514 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:19,514 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:19,514 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:19,514 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:19,515 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,515 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,515 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,515 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,517 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:19,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:19,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:19,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:19,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:19,518 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:19,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,519 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:19,519 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:19,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:19,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:19,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:19,619 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:19,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:19,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:19,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:19,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:19,621 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:19,621 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:19,621 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:19,621 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:19,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,622 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:19,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:19,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:19,623 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,623 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:19,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:19,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:19,623 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:19,624 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:19,624 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:19,624 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:19,624 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:19,624 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:19,624 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:19,624 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:19,624 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:19,624 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:19,624 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:19,624 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,624 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:19,624 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:19,625 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,625 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,625 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,625 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:19,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@519ada20[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:19,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:19,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,627 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,627 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:19,627 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,627 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:19,627 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:19,627 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:19,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:19,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:19,628 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:19,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:19,630 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:19,630 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:19,630 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:19,630 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:19,630 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:19,630 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,631 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,631 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,631 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,632 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,635 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:19,635 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:19,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:19,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:19,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:19,635 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:19,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:19,635 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:19,635 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,636 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:19,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:19,636 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,636 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,636 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,636 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,636 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:19,636 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:19,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:19,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893099652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,716 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/dc958cfbb36647bcaf9731ab0f733cb5 2023-03-27 04:57:19,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:19,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42562 deadline: 1679893099733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,737 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:19,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:19,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:19,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:19,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:19,739 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:19,739 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:19,739 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:19,739 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:19,739 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,739 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,740 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,740 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:19,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:19,740 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,740 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:19,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:19,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:19,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:19,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:19,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:19,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:19,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:19,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:19,741 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:19,741 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:19,741 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:19,741 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:19,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:19,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:19,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:19,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:19,742 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,742 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,742 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,742 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:19,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@200d3148[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:19,743 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:19,744 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,744 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,744 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:19,744 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:19,744 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,744 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:19,744 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:19,745 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,745 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,745 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:19,745 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:19,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:19,747 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:19,747 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:19,747 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:19,747 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:19,747 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:19,748 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,748 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,748 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:19,748 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,749 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,751 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,751 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,751 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:19,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:19,751 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,751 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,751 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:19,751 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:19,751 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,751 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:19,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:19,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:19,751 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:19,751 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,751 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:19,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:19,851 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:19,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:19,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:19,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:19,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:19,854 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:19,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:19,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:19,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:19,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,855 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,855 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:19,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:19,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:19,855 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,855 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:19,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:19,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:19,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:19,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:19,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:19,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:19,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:19,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:19,856 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:19,856 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:19,856 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:19,856 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:19,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:19,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:19,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:19,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:19,857 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:19,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fc658ea[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:19,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:19,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,859 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,859 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:19,859 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,859 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:19,859 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:19,859 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:19,860 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,860 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,860 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:19,860 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:19,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:19,861 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:19,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:19,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:19,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:19,862 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:19,862 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,862 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,862 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:19,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:19,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,864 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:19,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:19,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:19,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:19,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:19,865 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:19,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:19,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,865 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:19,865 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:19,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:19,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,866 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,866 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,866 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,866 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:19,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893099956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,965 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:19,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:19,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:19,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:19,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:19,968 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:19,968 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:19,968 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:19,968 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:19,968 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,968 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,969 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:19,969 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:19,969 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,969 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:19,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:19,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:19,970 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:19,970 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:19,970 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:19,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:19,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:19,971 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:19,971 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:19,971 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:19,971 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:19,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:19,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:19,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:19,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:19,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,972 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:19,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@109a5a54[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:19,973 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:19,973 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,974 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,974 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:19,974 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,974 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:19,974 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:19,974 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:19,975 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,975 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:19,975 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,975 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:19,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:19,976 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:19,976 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:19,976 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:19,976 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:19,976 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:19,977 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,977 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:19,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,977 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:19,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:19,978 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,979 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,980 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:19,980 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,980 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:19,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:19,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:19,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:19,980 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:19,980 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:19,981 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:19,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:19,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:19,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:19,981 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:19,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:19,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:20,081 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:20,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:20,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:20,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:20,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:20,083 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:20,083 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:20,083 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:20,083 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:20,084 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,084 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:20,085 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:20,085 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:20,085 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,085 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:20,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:20,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:20,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:20,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:20,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:20,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:20,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:20,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:20,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:20,086 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:20,086 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:20,086 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:20,086 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:20,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:20,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:20,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,087 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:20,087 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:20,087 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,087 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:20,087 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:20,087 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:20,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:20,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1756c9bd[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:20,089 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:20,089 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:20,089 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:20,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:20,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:20,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:20,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:20,090 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,090 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:20,090 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:20,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:20,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:20,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:20,094 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:20,094 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:20,094 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:20,094 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:20,094 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:20,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:20,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:20,096 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:20,098 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:20,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:20,100 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:20,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:20,100 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,100 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:20,100 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:20,100 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:20,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:20,100 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:20,100 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:20,100 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,100 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:20,100 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:20,100 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:20,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:20,124 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/4e9a034018d240acb20b65f5a84f6b8f 2023-03-27 04:57:20,140 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/b0be9ce904974ab49dd01782ad215306 2023-03-27 04:57:20,144 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/dc958cfbb36647bcaf9731ab0f733cb5 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/dc958cfbb36647bcaf9731ab0f733cb5 2023-03-27 04:57:20,148 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/dc958cfbb36647bcaf9731ab0f733cb5, entries=150, sequenceid=351, filesize=11.9 K 2023-03-27 04:57:20,148 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/4e9a034018d240acb20b65f5a84f6b8f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/4e9a034018d240acb20b65f5a84f6b8f 2023-03-27 04:57:20,152 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/4e9a034018d240acb20b65f5a84f6b8f, entries=150, sequenceid=351, filesize=11.9 K 2023-03-27 04:57:20,154 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/b0be9ce904974ab49dd01782ad215306 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/b0be9ce904974ab49dd01782ad215306 2023-03-27 04:57:20,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/b0be9ce904974ab49dd01782ad215306, entries=150, sequenceid=351, filesize=11.9 K 2023-03-27 04:57:20,161 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c239f1ca264c8e6332ec629bacfc12b1 in 853ms, sequenceid=351, compaction requested=true 2023-03-27 04:57:20,161 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:20,161 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:20,161 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:20,162 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:20,161 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:20,162 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:20,162 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:20,163 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/A is initiating minor compaction (all files) 2023-03-27 04:57:20,163 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:20,163 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/A in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:20,163 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/B is initiating minor compaction (all files) 2023-03-27 04:57:20,163 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/4ea4715684ba4d639d89a207d3c010e5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/b57acd8f9c11402bad2d401c26e2074a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/dc958cfbb36647bcaf9731ab0f733cb5] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=36.4 K 2023-03-27 04:57:20,163 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/B in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:20,163 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/7c3f80e937604805b8fcd03545104433, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/f40e3039f90d4a8c990e6da0618dce5f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/4e9a034018d240acb20b65f5a84f6b8f] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=36.4 K 2023-03-27 04:57:20,163 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 4ea4715684ba4d639d89a207d3c010e5, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1679893037519 2023-03-27 04:57:20,163 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 7c3f80e937604805b8fcd03545104433, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1679893037519 2023-03-27 04:57:20,163 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting b57acd8f9c11402bad2d401c26e2074a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1679893037602 2023-03-27 04:57:20,164 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting f40e3039f90d4a8c990e6da0618dce5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1679893037602 2023-03-27 04:57:20,164 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting dc958cfbb36647bcaf9731ab0f733cb5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1679893038686 2023-03-27 04:57:20,164 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 4e9a034018d240acb20b65f5a84f6b8f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1679893038686 2023-03-27 04:57:20,169 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#A#compaction#277 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:20,172 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#B#compaction#278 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:20,201 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:20,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:20,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:20,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:20,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:20,203 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:20,203 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:20,203 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:20,203 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:20,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:20,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,204 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:20,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:20,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:20,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,205 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:20,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:20,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:20,205 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:20,205 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:20,205 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:20,205 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:20,205 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:20,205 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:20,205 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:20,206 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:20,206 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:20,206 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-27 04:57:20,206 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:20,206 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:20,206 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:20,206 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:20,206 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:20,206 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:20,214 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/0a2d0b62920d445b8e39592b6621f11d 2023-03-27 04:57:20,224 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/cc1072e7465a4c5892932bc3fd38273f 2023-03-27 04:57:20,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:20,462 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:20,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:20,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893100484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,582 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/0fa546eb179340eab9961f022122ae60 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/0fa546eb179340eab9961f022122ae60 2023-03-27 04:57:20,583 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/6c6f315bb84748e9abf46fd11fdf3a50 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/6c6f315bb84748e9abf46fd11fdf3a50 2023-03-27 04:57:20,586 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/A of c239f1ca264c8e6332ec629bacfc12b1 into 0fa546eb179340eab9961f022122ae60(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:20,586 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:20,586 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/A, priority=13, startTime=1679893040161; duration=0sec 2023-03-27 04:57:20,586 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:20,586 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:20,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:20,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893100588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,594 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:20,595 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/C is initiating minor compaction (all files) 2023-03-27 04:57:20,595 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/C in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:20,595 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9813b20303744ebf9340988b1c1cb293, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/1f59565948a34c81bbf18981fd3865b7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/b0be9ce904974ab49dd01782ad215306] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=36.4 K 2023-03-27 04:57:20,595 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 9813b20303744ebf9340988b1c1cb293, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1679893037519 2023-03-27 04:57:20,595 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 1f59565948a34c81bbf18981fd3865b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1679893037602 2023-03-27 04:57:20,595 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting b0be9ce904974ab49dd01782ad215306, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1679893038686 2023-03-27 04:57:20,597 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/B of c239f1ca264c8e6332ec629bacfc12b1 into 6c6f315bb84748e9abf46fd11fdf3a50(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:20,597 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:20,597 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/B, priority=13, startTime=1679893040161; duration=0sec 2023-03-27 04:57:20,597 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:20,601 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#C#compaction#282 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:20,638 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/56f2aa0f25394f9aa6b7905a4b81d080 2023-03-27 04:57:20,642 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/0a2d0b62920d445b8e39592b6621f11d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/0a2d0b62920d445b8e39592b6621f11d 2023-03-27 04:57:20,647 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/0a2d0b62920d445b8e39592b6621f11d, entries=150, sequenceid=371, filesize=11.9 K 2023-03-27 04:57:20,648 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/cc1072e7465a4c5892932bc3fd38273f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/cc1072e7465a4c5892932bc3fd38273f 2023-03-27 04:57:20,651 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/cc1072e7465a4c5892932bc3fd38273f, entries=150, sequenceid=371, filesize=11.9 K 2023-03-27 04:57:20,652 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/56f2aa0f25394f9aa6b7905a4b81d080 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/56f2aa0f25394f9aa6b7905a4b81d080 2023-03-27 04:57:20,654 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/56f2aa0f25394f9aa6b7905a4b81d080, entries=150, sequenceid=371, filesize=11.9 K 2023-03-27 04:57:20,655 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for c239f1ca264c8e6332ec629bacfc12b1 in 449ms, sequenceid=371, compaction requested=false 2023-03-27 04:57:20,655 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:20,655 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:20,655 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:20,655 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:20,655 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,655 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:20,655 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:20,656 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,656 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:20,657 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:20,657 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:20,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:20,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e182d6e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:20,658 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:20,658 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:20,658 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:20,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:20,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:20,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:20,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:20,659 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,659 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:20,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:20,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:20,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:20,659 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:20,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:20,661 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:20,661 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:20,661 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:20,661 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:20,661 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:20,662 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:20,662 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,663 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:20,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:20,663 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:20,664 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:20,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:20,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:20,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:20,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:20,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:20,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,665 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:20,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:20,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:20,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:20,666 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:20,666 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,666 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:20,666 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:20,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:20,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:20,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:20,766 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:20,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:20,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:20,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:20,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:20,769 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:20,769 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:20,769 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:20,769 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:20,769 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,769 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:20,770 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:20,770 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:20,770 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,770 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:20,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:20,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:20,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:20,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:20,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:20,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:20,771 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:20,771 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:20,771 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:20,771 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:20,772 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:20,772 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:20,772 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:20,772 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-27 04:57:20,772 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:20,772 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:20,772 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:20,772 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:20,772 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:20,772 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:20,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:20,793 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:20,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:20,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893100837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:20,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:20,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893100939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:21,008 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/8f43d4ecab464f139b568e65a1676804 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/8f43d4ecab464f139b568e65a1676804 2023-03-27 04:57:21,012 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/C of c239f1ca264c8e6332ec629bacfc12b1 into 8f43d4ecab464f139b568e65a1676804(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:21,012 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:21,012 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/C, priority=13, startTime=1679893040162; duration=0sec 2023-03-27 04:57:21,013 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:21,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:21,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893101142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:21,179 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/50f56ad5b9404dcea6260ac10c21f991 2023-03-27 04:57:21,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:21,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893101446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:21,587 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/e73313e0dd4947398091eefddc811795 2023-03-27 04:57:21,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:21,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42562 deadline: 1679893101746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:21,749 DEBUG [Thread-1662] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., hostname=jenkins-hbase20.apache.org,44923,1679892955505, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:57:21,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:21,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893101949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:21,995 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/13f429006df2464f9026eac27ed51da6 2023-03-27 04:57:21,999 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/50f56ad5b9404dcea6260ac10c21f991 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/50f56ad5b9404dcea6260ac10c21f991 2023-03-27 04:57:22,002 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/50f56ad5b9404dcea6260ac10c21f991, entries=150, sequenceid=385, filesize=11.9 K 2023-03-27 04:57:22,003 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/e73313e0dd4947398091eefddc811795 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/e73313e0dd4947398091eefddc811795 2023-03-27 04:57:22,006 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/e73313e0dd4947398091eefddc811795, entries=150, sequenceid=385, filesize=11.9 K 2023-03-27 04:57:22,007 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/13f429006df2464f9026eac27ed51da6 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/13f429006df2464f9026eac27ed51da6 2023-03-27 04:57:22,010 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/13f429006df2464f9026eac27ed51da6, entries=150, sequenceid=385, filesize=11.9 K 2023-03-27 04:57:22,011 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c239f1ca264c8e6332ec629bacfc12b1 in 1239ms, sequenceid=385, compaction requested=true 2023-03-27 04:57:22,011 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:22,011 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:22,011 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:22,011 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:22,011 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,011 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:22,011 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:22,012 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,012 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:22,012 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:22,012 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:22,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:22,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c42f4f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:22,014 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:22,014 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:22,014 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:22,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:22,014 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:22,014 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,014 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:22,014 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:22,014 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:22,015 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:22,015 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:22,015 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,015 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:22,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:22,017 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:22,017 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:22,017 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:22,017 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:22,017 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:22,018 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:22,018 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:22,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:22,018 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:22,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:22,019 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:22,020 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,021 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,021 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:22,021 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:22,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:22,021 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:22,021 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:22,022 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:22,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:22,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:22,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:22,022 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,022 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:22,022 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:22,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:22,022 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:22,022 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:22,022 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:22,122 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:22,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:22,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:22,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:22,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:22,125 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:22,125 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:22,125 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:22,125 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:22,125 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:22,125 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:22,126 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:22,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:22,126 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:22,126 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,126 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:22,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:22,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:22,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:22,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:22,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:22,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:22,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:22,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:22,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:22,127 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:22,127 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. started... 2023-03-27 04:57:22,127 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-27 04:57:22,127 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:22,128 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:22,128 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:22,128 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:22,128 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:22,128 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:22,534 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/4c2989c9fd364043ae88a1cbe3663281 2023-03-27 04:57:22,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:22,802 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. as already flushing 2023-03-27 04:57:22,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:22,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42590 deadline: 1679893102814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:22,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42610 deadline: 1679893102815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:22,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42556 deadline: 1679893102816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:22,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42590 deadline: 1679893102917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:22,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42610 deadline: 1679893102918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:22,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42556 deadline: 1679893102918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:22,942 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/a2cd2b2c22e64bdb938e0e0c6d842baa 2023-03-27 04:57:22,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:22,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42592 deadline: 1679893102950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,090 DEBUG [Thread-1677] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x509adaa8 to 127.0.0.1:61059 2023-03-27 04:57:23,090 DEBUG [Thread-1669] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36523181 to 127.0.0.1:61059 2023-03-27 04:57:23,090 DEBUG [Thread-1677] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:23,090 DEBUG [Thread-1669] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:23,090 DEBUG [Thread-1675] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5654f8ca to 127.0.0.1:61059 2023-03-27 04:57:23,091 DEBUG [Thread-1675] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:23,092 DEBUG [Thread-1671] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b458e37 to 127.0.0.1:61059 2023-03-27 04:57:23,092 DEBUG [Thread-1671] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:23,093 DEBUG [Thread-1673] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bae1a64 to 127.0.0.1:61059 2023-03-27 04:57:23,093 DEBUG [Thread-1673] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:23,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:23,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42590 deadline: 1679893103119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:23,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42610 deadline: 1679893103122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:23,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42556 deadline: 1679893103122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,350 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/668cb90470624e3f85d59ed93aeff70d 2023-03-27 04:57:23,360 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/4c2989c9fd364043ae88a1cbe3663281 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/4c2989c9fd364043ae88a1cbe3663281 2023-03-27 04:57:23,364 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/4c2989c9fd364043ae88a1cbe3663281, entries=150, sequenceid=406, filesize=11.9 K 2023-03-27 04:57:23,365 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/a2cd2b2c22e64bdb938e0e0c6d842baa as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/a2cd2b2c22e64bdb938e0e0c6d842baa 2023-03-27 04:57:23,370 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/a2cd2b2c22e64bdb938e0e0c6d842baa, entries=150, sequenceid=406, filesize=11.9 K 2023-03-27 04:57:23,371 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/668cb90470624e3f85d59ed93aeff70d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/668cb90470624e3f85d59ed93aeff70d 2023-03-27 04:57:23,375 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/668cb90470624e3f85d59ed93aeff70d, entries=150, sequenceid=406, filesize=11.9 K 2023-03-27 04:57:23,376 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for c239f1ca264c8e6332ec629bacfc12b1 in 1249ms, sequenceid=406, compaction requested=true 2023-03-27 04:57:23,376 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:23,376 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:23,376 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:23,376 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:23,376 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,376 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:23,376 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:23,378 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,378 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:23,379 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:23,379 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:23,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:23,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:23,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@414e8d99[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:23,380 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:23,380 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:23,381 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:23,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:23,381 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:23,381 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:23,381 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:23,381 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,382 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:23,382 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,382 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:23,382 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:23,382 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:23,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:23,385 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:23,385 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:23,385 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:23,385 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:23,385 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:23,386 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:23,386 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:23,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:23,387 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:23,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:23,387 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:23,389 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,391 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,391 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:23,391 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:23,391 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:23,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:23,392 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:23,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:23,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:23,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:23,391 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:23,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:23,392 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:23,393 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:23,393 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:23,393 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:23,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:23,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:23,393 DEBUG [Thread-1668] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:23,394 DEBUG [Thread-1668] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:23,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:23,424 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-27 04:57:23,424 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:23,424 DEBUG [Thread-1664] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x407b9cb0 to 127.0.0.1:61059 2023-03-27 04:57:23,424 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:23,424 DEBUG [Thread-1664] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:23,424 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:23,425 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:23,425 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:23,425 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:23,428 DEBUG [Thread-1658] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d3f4b11 to 127.0.0.1:61059 2023-03-27 04:57:23,428 DEBUG [Thread-1660] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7187fcf3 to 127.0.0.1:61059 2023-03-27 04:57:23,429 DEBUG [Thread-1658] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:23,429 DEBUG [Thread-1660] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:23,494 DEBUG [Thread-1668] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:23,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:23,835 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/9ba76962d05e448a9c27670f299b75ed 2023-03-27 04:57:24,254 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/65b8ceb789274d50820464ab9939c3ed 2023-03-27 04:57:24,668 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/478be351740b4539a5f38451cde94b1e 2023-03-27 04:57:24,672 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/9ba76962d05e448a9c27670f299b75ed as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/9ba76962d05e448a9c27670f299b75ed 2023-03-27 04:57:24,675 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/9ba76962d05e448a9c27670f299b75ed, entries=150, sequenceid=419, filesize=11.9 K 2023-03-27 04:57:24,676 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/65b8ceb789274d50820464ab9939c3ed as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/65b8ceb789274d50820464ab9939c3ed 2023-03-27 04:57:24,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/65b8ceb789274d50820464ab9939c3ed, entries=150, sequenceid=419, filesize=11.9 K 2023-03-27 04:57:24,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/478be351740b4539a5f38451cde94b1e as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/478be351740b4539a5f38451cde94b1e 2023-03-27 04:57:24,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/478be351740b4539a5f38451cde94b1e, entries=150, sequenceid=419, filesize=11.9 K 2023-03-27 04:57:24,684 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=13.42 KB/13740 for c239f1ca264c8e6332ec629bacfc12b1 in 1261ms, sequenceid=419, compaction requested=true 2023-03-27 04:57:24,684 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:24,685 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:24,685 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-27 04:57:24,685 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-27 04:57:24,685 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:24,685 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:24,685 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-27 04:57:24,686 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/A is initiating minor compaction (all files) 2023-03-27 04:57:24,686 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-27 04:57:24,686 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/A in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:24,686 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/B is initiating minor compaction (all files) 2023-03-27 04:57:24,686 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/0fa546eb179340eab9961f022122ae60, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/0a2d0b62920d445b8e39592b6621f11d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/50f56ad5b9404dcea6260ac10c21f991, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/4c2989c9fd364043ae88a1cbe3663281, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/9ba76962d05e448a9c27670f299b75ed] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=60.3 K 2023-03-27 04:57:24,686 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/B in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:24,686 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/6c6f315bb84748e9abf46fd11fdf3a50, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/cc1072e7465a4c5892932bc3fd38273f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/e73313e0dd4947398091eefddc811795, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/a2cd2b2c22e64bdb938e0e0c6d842baa, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/65b8ceb789274d50820464ab9939c3ed] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=60.3 K 2023-03-27 04:57:24,686 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 0fa546eb179340eab9961f022122ae60, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1679893038686 2023-03-27 04:57:24,686 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 6c6f315bb84748e9abf46fd11fdf3a50, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1679893038686 2023-03-27 04:57:24,686 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 0a2d0b62920d445b8e39592b6621f11d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1679893039336 2023-03-27 04:57:24,686 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting cc1072e7465a4c5892932bc3fd38273f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1679893039336 2023-03-27 04:57:24,686 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 50f56ad5b9404dcea6260ac10c21f991, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1679893040478 2023-03-27 04:57:24,686 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting e73313e0dd4947398091eefddc811795, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1679893040478 2023-03-27 04:57:24,687 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 4c2989c9fd364043ae88a1cbe3663281, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1679893040827 2023-03-27 04:57:24,687 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting a2cd2b2c22e64bdb938e0e0c6d842baa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1679893040827 2023-03-27 04:57:24,687 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 9ba76962d05e448a9c27670f299b75ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1679893042813 2023-03-27 04:57:24,687 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 65b8ceb789274d50820464ab9939c3ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1679893042813 2023-03-27 04:57:24,696 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#A#compaction#292 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:24,700 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#B#compaction#293 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:24,709 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/015ff948151d472cbbe65231689881fa as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/015ff948151d472cbbe65231689881fa 2023-03-27 04:57:24,710 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/2cd0a4b434554d9993f1cd92fd8423a7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/2cd0a4b434554d9993f1cd92fd8423a7 2023-03-27 04:57:24,712 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/A of c239f1ca264c8e6332ec629bacfc12b1 into 015ff948151d472cbbe65231689881fa(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:24,712 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:24,712 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/A, priority=11, startTime=1679893044685; duration=0sec 2023-03-27 04:57:24,712 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:24,712 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-27 04:57:24,713 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/B of c239f1ca264c8e6332ec629bacfc12b1 into 2cd0a4b434554d9993f1cd92fd8423a7(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:24,713 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:24,713 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/B, priority=11, startTime=1679893044685; duration=0sec 2023-03-27 04:57:24,713 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:24,713 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-27 04:57:24,713 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): c239f1ca264c8e6332ec629bacfc12b1/C is initiating minor compaction (all files) 2023-03-27 04:57:24,713 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c239f1ca264c8e6332ec629bacfc12b1/C in TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:24,713 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/8f43d4ecab464f139b568e65a1676804, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/56f2aa0f25394f9aa6b7905a4b81d080, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/13f429006df2464f9026eac27ed51da6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/668cb90470624e3f85d59ed93aeff70d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/478be351740b4539a5f38451cde94b1e] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp, totalSize=60.3 K 2023-03-27 04:57:24,714 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 8f43d4ecab464f139b568e65a1676804, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1679893038686 2023-03-27 04:57:24,714 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 56f2aa0f25394f9aa6b7905a4b81d080, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1679893039336 2023-03-27 04:57:24,714 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 13f429006df2464f9026eac27ed51da6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1679893040478 2023-03-27 04:57:24,714 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 668cb90470624e3f85d59ed93aeff70d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1679893040827 2023-03-27 04:57:24,715 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 478be351740b4539a5f38451cde94b1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1679893042813 2023-03-27 04:57:24,721 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): c239f1ca264c8e6332ec629bacfc12b1#C#compaction#294 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:24,965 DEBUG [Thread-1666] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f456fa2 to 127.0.0.1:61059 2023-03-27 04:57:24,965 DEBUG [Thread-1666] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:25,135 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/ddb1bdfb20db4a3b9a471d639bbf2777 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/ddb1bdfb20db4a3b9a471d639bbf2777 2023-03-27 04:57:25,142 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c239f1ca264c8e6332ec629bacfc12b1/C of c239f1ca264c8e6332ec629bacfc12b1 into ddb1bdfb20db4a3b9a471d639bbf2777(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:25,142 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:25,142 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1., storeName=c239f1ca264c8e6332ec629bacfc12b1/C, priority=11, startTime=1679893044685; duration=0sec 2023-03-27 04:57:25,143 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:25,786 DEBUG [Thread-1662] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x712d78ca to 127.0.0.1:61059 2023-03-27 04:57:25,786 DEBUG [Thread-1662] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:25,786 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-27 04:57:25,786 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 41 2023-03-27 04:57:25,787 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 41 2023-03-27 04:57:25,787 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 82 2023-03-27 04:57:25,787 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 46 2023-03-27 04:57:25,787 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 122 2023-03-27 04:57:25,787 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-27 04:57:25,787 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(398): read 6079 2023-03-27 04:57:25,787 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(398): read 5935 2023-03-27 04:57:25,787 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(398): read 5912 2023-03-27 04:57:25,787 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(398): read 6025 2023-03-27 04:57:25,787 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(398): read 5914 2023-03-27 04:57:25,787 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-27 04:57:25,787 INFO [Listener at localhost.localdomain/43311] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-27 04:57:25,788 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15a0890b to 127.0.0.1:61059 2023-03-27 04:57:25,788 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:25,788 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-27 04:57:25,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-27 04:57:25,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:25,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-27 04:57:25,794 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893045794"}]},"ts":"1679893045794"} 2023-03-27 04:57:25,795 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-27 04:57:25,801 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-27 04:57:25,802 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c239f1ca264c8e6332ec629bacfc12b1, UNASSIGN}] 2023-03-27 04:57:25,803 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c239f1ca264c8e6332ec629bacfc12b1, UNASSIGN 2023-03-27 04:57:25,804 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=c239f1ca264c8e6332ec629bacfc12b1, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:25,804 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893045804"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679893045804"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679893045804"}]},"ts":"1679893045804"} 2023-03-27 04:57:25,805 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:57:25,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-27 04:57:25,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:25,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing c239f1ca264c8e6332ec629bacfc12b1, disabling compactions & flushes 2023-03-27 04:57:25,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:25,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:25,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. after waiting 0 ms 2023-03-27 04:57:25,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:25,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing c239f1ca264c8e6332ec629bacfc12b1 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-27 04:57:25,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=A 2023-03-27 04:57:25,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:25,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=B 2023-03-27 04:57:25,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:25,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c239f1ca264c8e6332ec629bacfc12b1, store=C 2023-03-27 04:57:25,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:26,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-27 04:57:26,367 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/ed981eecf0d444179f98fe00114f6601 2023-03-27 04:57:26,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-27 04:57:26,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/8b13269f273b4fc582659958002622ce 2023-03-27 04:57:26,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-27 04:57:27,204 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/4dc0537342864498a8ec39acba53982c 2023-03-27 04:57:27,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/A/ed981eecf0d444179f98fe00114f6601 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ed981eecf0d444179f98fe00114f6601 2023-03-27 04:57:27,215 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ed981eecf0d444179f98fe00114f6601, entries=100, sequenceid=429, filesize=9.5 K 2023-03-27 04:57:27,216 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/B/8b13269f273b4fc582659958002622ce as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/8b13269f273b4fc582659958002622ce 2023-03-27 04:57:27,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/8b13269f273b4fc582659958002622ce, entries=100, sequenceid=429, filesize=9.5 K 2023-03-27 04:57:27,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/.tmp/C/4dc0537342864498a8ec39acba53982c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/4dc0537342864498a8ec39acba53982c 2023-03-27 04:57:27,226 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/4dc0537342864498a8ec39acba53982c, entries=100, sequenceid=429, filesize=9.5 K 2023-03-27 04:57:27,227 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for c239f1ca264c8e6332ec629bacfc12b1 in 1269ms, sequenceid=429, compaction requested=false 2023-03-27 04:57:27,243 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/f3601d643cf14e038ec451d93ced9d11, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/1db194bfa3404a0d9e4a782ffa5a2c1b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ec53091b9cec43119a2420b2b7bcf08c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/a5bff4b34f1a45fab83e14d45abd074a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8e813c9831164932a2a1376d69ea559c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ddc10a0be8604f31abe8df7361fc018b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ddad6ec5c1194d398382544242552fad, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/5833f1846ebd4f0aacae66e6772b796d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/56555dfa88194654ba7fce1a2c8f9f57, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8d89fbcc896a404abaea45eb3a193209, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/3c2e88e8560e4675aefd7755e6257c08, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/e99886582c2746a0a68c8b7682bba8c8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/82650da2334046ef974fc6510e6a4e7a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8f6520116ee7455eb54bd5716edd76b6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ad49fcdd8bbd46bda4b1b63628cbade1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/386bd6a96b0541c68a48994f9eb06fd1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/76c264eb0b14472a88e7c75cd476706b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/4de2b01144b843298165307a796a1418, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/638391d2e3dd4ba98ca8592c04205492, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/16a6959672094ed69bda02fa50a484de, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/f27366f115494d02964f5c836731a940, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/4ea4715684ba4d639d89a207d3c010e5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/b57acd8f9c11402bad2d401c26e2074a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/0fa546eb179340eab9961f022122ae60, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/dc958cfbb36647bcaf9731ab0f733cb5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/0a2d0b62920d445b8e39592b6621f11d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/50f56ad5b9404dcea6260ac10c21f991, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/4c2989c9fd364043ae88a1cbe3663281, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/9ba76962d05e448a9c27670f299b75ed] to archive 2023-03-27 04:57:27,244 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:57:27,245 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/f3601d643cf14e038ec451d93ced9d11 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/f3601d643cf14e038ec451d93ced9d11 2023-03-27 04:57:27,246 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/1db194bfa3404a0d9e4a782ffa5a2c1b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/1db194bfa3404a0d9e4a782ffa5a2c1b 2023-03-27 04:57:27,247 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ec53091b9cec43119a2420b2b7bcf08c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ec53091b9cec43119a2420b2b7bcf08c 2023-03-27 04:57:27,248 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/a5bff4b34f1a45fab83e14d45abd074a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/a5bff4b34f1a45fab83e14d45abd074a 2023-03-27 04:57:27,249 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8e813c9831164932a2a1376d69ea559c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8e813c9831164932a2a1376d69ea559c 2023-03-27 04:57:27,250 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ddc10a0be8604f31abe8df7361fc018b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ddc10a0be8604f31abe8df7361fc018b 2023-03-27 04:57:27,250 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ddad6ec5c1194d398382544242552fad to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ddad6ec5c1194d398382544242552fad 2023-03-27 04:57:27,251 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/5833f1846ebd4f0aacae66e6772b796d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/5833f1846ebd4f0aacae66e6772b796d 2023-03-27 04:57:27,252 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/56555dfa88194654ba7fce1a2c8f9f57 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/56555dfa88194654ba7fce1a2c8f9f57 2023-03-27 04:57:27,253 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8d89fbcc896a404abaea45eb3a193209 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8d89fbcc896a404abaea45eb3a193209 2023-03-27 04:57:27,254 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/3c2e88e8560e4675aefd7755e6257c08 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/3c2e88e8560e4675aefd7755e6257c08 2023-03-27 04:57:27,255 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/e99886582c2746a0a68c8b7682bba8c8 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/e99886582c2746a0a68c8b7682bba8c8 2023-03-27 04:57:27,256 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/82650da2334046ef974fc6510e6a4e7a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/82650da2334046ef974fc6510e6a4e7a 2023-03-27 04:57:27,257 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8f6520116ee7455eb54bd5716edd76b6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/8f6520116ee7455eb54bd5716edd76b6 2023-03-27 04:57:27,258 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ad49fcdd8bbd46bda4b1b63628cbade1 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ad49fcdd8bbd46bda4b1b63628cbade1 2023-03-27 04:57:27,260 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/386bd6a96b0541c68a48994f9eb06fd1 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/386bd6a96b0541c68a48994f9eb06fd1 2023-03-27 04:57:27,261 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/76c264eb0b14472a88e7c75cd476706b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/76c264eb0b14472a88e7c75cd476706b 2023-03-27 04:57:27,262 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/4de2b01144b843298165307a796a1418 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/4de2b01144b843298165307a796a1418 2023-03-27 04:57:27,263 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/638391d2e3dd4ba98ca8592c04205492 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/638391d2e3dd4ba98ca8592c04205492 2023-03-27 04:57:27,265 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/16a6959672094ed69bda02fa50a484de to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/16a6959672094ed69bda02fa50a484de 2023-03-27 04:57:27,266 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/f27366f115494d02964f5c836731a940 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/f27366f115494d02964f5c836731a940 2023-03-27 04:57:27,267 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/4ea4715684ba4d639d89a207d3c010e5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/4ea4715684ba4d639d89a207d3c010e5 2023-03-27 04:57:27,269 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/b57acd8f9c11402bad2d401c26e2074a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/b57acd8f9c11402bad2d401c26e2074a 2023-03-27 04:57:27,270 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/0fa546eb179340eab9961f022122ae60 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/0fa546eb179340eab9961f022122ae60 2023-03-27 04:57:27,272 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/dc958cfbb36647bcaf9731ab0f733cb5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/dc958cfbb36647bcaf9731ab0f733cb5 2023-03-27 04:57:27,273 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/0a2d0b62920d445b8e39592b6621f11d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/0a2d0b62920d445b8e39592b6621f11d 2023-03-27 04:57:27,275 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/50f56ad5b9404dcea6260ac10c21f991 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/50f56ad5b9404dcea6260ac10c21f991 2023-03-27 04:57:27,276 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/4c2989c9fd364043ae88a1cbe3663281 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/4c2989c9fd364043ae88a1cbe3663281 2023-03-27 04:57:27,278 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/9ba76962d05e448a9c27670f299b75ed to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/9ba76962d05e448a9c27670f299b75ed 2023-03-27 04:57:27,282 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/2b78d509c97845c6a9fb76d825e50b22, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/916d986e286b451db619194a79c9711d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/b845f2c4122a4f6f8e669b9b49465931, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/1f92995c526e4806ad60265a35be82ca, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/2881c63986184959bc376482d796d161, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/4d2f403efa67460980f892e181148204, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/db75863e346a4a5b97d175e0af356831, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/38d3cc61e8eb4aab915c9aeefa597ed2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/5ee666f666a04f67a370a40b7c54ebff, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/ac1a9458cc614813b35160f21ccee3e9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/90ac56cb61b6413ebae4cab4833f53f5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/bb62d6711ab7497cb64a7e28cd83045f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/25d39513b9c6424fa7923177cecf8fce, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/ef69816aadb14c869d11d055ca5213a9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/116201ea78544a63bdad492a020fe72d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/bca1cdf6c02941809a8448ee9f30715a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/d6ca8a9f0a134be2b87e140545d88e02, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/be44783c9cc94c7295b833f78ff1be6e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/b3983c9980cb4c14914f908a2f323f3d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/b47e532304584ef8bd25be819d4af10c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/7c3f80e937604805b8fcd03545104433, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/d791984b08974d8686afb6a426dc5dee, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/f40e3039f90d4a8c990e6da0618dce5f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/6c6f315bb84748e9abf46fd11fdf3a50, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/4e9a034018d240acb20b65f5a84f6b8f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/cc1072e7465a4c5892932bc3fd38273f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/e73313e0dd4947398091eefddc811795, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/a2cd2b2c22e64bdb938e0e0c6d842baa, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/65b8ceb789274d50820464ab9939c3ed] to archive 2023-03-27 04:57:27,283 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:57:27,285 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/2b78d509c97845c6a9fb76d825e50b22 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/2b78d509c97845c6a9fb76d825e50b22 2023-03-27 04:57:27,287 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/916d986e286b451db619194a79c9711d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/916d986e286b451db619194a79c9711d 2023-03-27 04:57:27,288 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/b845f2c4122a4f6f8e669b9b49465931 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/b845f2c4122a4f6f8e669b9b49465931 2023-03-27 04:57:27,289 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/1f92995c526e4806ad60265a35be82ca to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/1f92995c526e4806ad60265a35be82ca 2023-03-27 04:57:27,290 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/2881c63986184959bc376482d796d161 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/2881c63986184959bc376482d796d161 2023-03-27 04:57:27,291 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/4d2f403efa67460980f892e181148204 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/4d2f403efa67460980f892e181148204 2023-03-27 04:57:27,292 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/db75863e346a4a5b97d175e0af356831 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/db75863e346a4a5b97d175e0af356831 2023-03-27 04:57:27,293 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/38d3cc61e8eb4aab915c9aeefa597ed2 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/38d3cc61e8eb4aab915c9aeefa597ed2 2023-03-27 04:57:27,295 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/5ee666f666a04f67a370a40b7c54ebff to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/5ee666f666a04f67a370a40b7c54ebff 2023-03-27 04:57:27,296 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/ac1a9458cc614813b35160f21ccee3e9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/ac1a9458cc614813b35160f21ccee3e9 2023-03-27 04:57:27,297 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/90ac56cb61b6413ebae4cab4833f53f5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/90ac56cb61b6413ebae4cab4833f53f5 2023-03-27 04:57:27,298 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/bb62d6711ab7497cb64a7e28cd83045f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/bb62d6711ab7497cb64a7e28cd83045f 2023-03-27 04:57:27,299 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/25d39513b9c6424fa7923177cecf8fce to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/25d39513b9c6424fa7923177cecf8fce 2023-03-27 04:57:27,300 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/ef69816aadb14c869d11d055ca5213a9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/ef69816aadb14c869d11d055ca5213a9 2023-03-27 04:57:27,300 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/116201ea78544a63bdad492a020fe72d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/116201ea78544a63bdad492a020fe72d 2023-03-27 04:57:27,301 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/bca1cdf6c02941809a8448ee9f30715a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/bca1cdf6c02941809a8448ee9f30715a 2023-03-27 04:57:27,302 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/d6ca8a9f0a134be2b87e140545d88e02 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/d6ca8a9f0a134be2b87e140545d88e02 2023-03-27 04:57:27,303 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/be44783c9cc94c7295b833f78ff1be6e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/be44783c9cc94c7295b833f78ff1be6e 2023-03-27 04:57:27,303 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/b3983c9980cb4c14914f908a2f323f3d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/b3983c9980cb4c14914f908a2f323f3d 2023-03-27 04:57:27,304 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/b47e532304584ef8bd25be819d4af10c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/b47e532304584ef8bd25be819d4af10c 2023-03-27 04:57:27,305 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/7c3f80e937604805b8fcd03545104433 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/7c3f80e937604805b8fcd03545104433 2023-03-27 04:57:27,305 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/d791984b08974d8686afb6a426dc5dee to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/d791984b08974d8686afb6a426dc5dee 2023-03-27 04:57:27,306 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/f40e3039f90d4a8c990e6da0618dce5f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/f40e3039f90d4a8c990e6da0618dce5f 2023-03-27 04:57:27,307 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/6c6f315bb84748e9abf46fd11fdf3a50 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/6c6f315bb84748e9abf46fd11fdf3a50 2023-03-27 04:57:27,308 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/4e9a034018d240acb20b65f5a84f6b8f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/4e9a034018d240acb20b65f5a84f6b8f 2023-03-27 04:57:27,308 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/cc1072e7465a4c5892932bc3fd38273f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/cc1072e7465a4c5892932bc3fd38273f 2023-03-27 04:57:27,309 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/e73313e0dd4947398091eefddc811795 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/e73313e0dd4947398091eefddc811795 2023-03-27 04:57:27,310 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/a2cd2b2c22e64bdb938e0e0c6d842baa to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/a2cd2b2c22e64bdb938e0e0c6d842baa 2023-03-27 04:57:27,310 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/65b8ceb789274d50820464ab9939c3ed to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/65b8ceb789274d50820464ab9939c3ed 2023-03-27 04:57:27,313 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/4c06ce6a33dc4399abbde618e62d74dc, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/ffa1842cf6c74d08ac909b7244465136, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/8637bf210c8e45cea98564c81a5761b8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/953c0410b9944b1bbd3ff3ed2423e2f7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/5975cb3a81d74afab8899bf00857330b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/d3fb6c005fc04179afda01407cf399d7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/260ee3377a6049649d69967a50e17c4f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/4b29091dd1dd45c69f80d506ee97ce66, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9638a68da76043b39fed2f0fcb245242, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9ac49fa75d6f41b9b00e416caf800146, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/d60035fa37ad4232b9bf5de2002f48f0, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/c22a903c4f714a599c7ee73e4f27a53c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/bd0770e0956a4e96914d9fd607752bb7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/30e9c317119b4121a1cd3e865ee84c51, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9115b7b467a44570ad80968861d3a9e2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/46b3c110088f46b3bf36edfa19b7cb10, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/e9298c70d8a5465f9f6d010330e86428, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/57771109568344f9b5aa7df32c1df13d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/deef1b23cba8495a839e0a24181baeb2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/0b434c51995b4c92a92798d30d45420a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9813b20303744ebf9340988b1c1cb293, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/677ab0204903443593a6315cfd301a94, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/1f59565948a34c81bbf18981fd3865b7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/8f43d4ecab464f139b568e65a1676804, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/b0be9ce904974ab49dd01782ad215306, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/56f2aa0f25394f9aa6b7905a4b81d080, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/13f429006df2464f9026eac27ed51da6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/668cb90470624e3f85d59ed93aeff70d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/478be351740b4539a5f38451cde94b1e] to archive 2023-03-27 04:57:27,313 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:57:27,314 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/4c06ce6a33dc4399abbde618e62d74dc to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/4c06ce6a33dc4399abbde618e62d74dc 2023-03-27 04:57:27,315 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/ffa1842cf6c74d08ac909b7244465136 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/ffa1842cf6c74d08ac909b7244465136 2023-03-27 04:57:27,316 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/8637bf210c8e45cea98564c81a5761b8 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/8637bf210c8e45cea98564c81a5761b8 2023-03-27 04:57:27,317 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/953c0410b9944b1bbd3ff3ed2423e2f7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/953c0410b9944b1bbd3ff3ed2423e2f7 2023-03-27 04:57:27,317 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/5975cb3a81d74afab8899bf00857330b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/5975cb3a81d74afab8899bf00857330b 2023-03-27 04:57:27,318 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/d3fb6c005fc04179afda01407cf399d7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/d3fb6c005fc04179afda01407cf399d7 2023-03-27 04:57:27,319 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/260ee3377a6049649d69967a50e17c4f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/260ee3377a6049649d69967a50e17c4f 2023-03-27 04:57:27,320 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/4b29091dd1dd45c69f80d506ee97ce66 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/4b29091dd1dd45c69f80d506ee97ce66 2023-03-27 04:57:27,320 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9638a68da76043b39fed2f0fcb245242 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9638a68da76043b39fed2f0fcb245242 2023-03-27 04:57:27,321 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9ac49fa75d6f41b9b00e416caf800146 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9ac49fa75d6f41b9b00e416caf800146 2023-03-27 04:57:27,322 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/d60035fa37ad4232b9bf5de2002f48f0 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/d60035fa37ad4232b9bf5de2002f48f0 2023-03-27 04:57:27,323 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/c22a903c4f714a599c7ee73e4f27a53c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/c22a903c4f714a599c7ee73e4f27a53c 2023-03-27 04:57:27,323 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/bd0770e0956a4e96914d9fd607752bb7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/bd0770e0956a4e96914d9fd607752bb7 2023-03-27 04:57:27,324 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/30e9c317119b4121a1cd3e865ee84c51 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/30e9c317119b4121a1cd3e865ee84c51 2023-03-27 04:57:27,325 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9115b7b467a44570ad80968861d3a9e2 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9115b7b467a44570ad80968861d3a9e2 2023-03-27 04:57:27,325 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/46b3c110088f46b3bf36edfa19b7cb10 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/46b3c110088f46b3bf36edfa19b7cb10 2023-03-27 04:57:27,326 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/e9298c70d8a5465f9f6d010330e86428 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/e9298c70d8a5465f9f6d010330e86428 2023-03-27 04:57:27,327 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/57771109568344f9b5aa7df32c1df13d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/57771109568344f9b5aa7df32c1df13d 2023-03-27 04:57:27,328 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/deef1b23cba8495a839e0a24181baeb2 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/deef1b23cba8495a839e0a24181baeb2 2023-03-27 04:57:27,329 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/0b434c51995b4c92a92798d30d45420a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/0b434c51995b4c92a92798d30d45420a 2023-03-27 04:57:27,330 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9813b20303744ebf9340988b1c1cb293 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/9813b20303744ebf9340988b1c1cb293 2023-03-27 04:57:27,331 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/677ab0204903443593a6315cfd301a94 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/677ab0204903443593a6315cfd301a94 2023-03-27 04:57:27,332 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/1f59565948a34c81bbf18981fd3865b7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/1f59565948a34c81bbf18981fd3865b7 2023-03-27 04:57:27,333 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/8f43d4ecab464f139b568e65a1676804 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/8f43d4ecab464f139b568e65a1676804 2023-03-27 04:57:27,334 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/b0be9ce904974ab49dd01782ad215306 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/b0be9ce904974ab49dd01782ad215306 2023-03-27 04:57:27,335 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/56f2aa0f25394f9aa6b7905a4b81d080 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/56f2aa0f25394f9aa6b7905a4b81d080 2023-03-27 04:57:27,336 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/13f429006df2464f9026eac27ed51da6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/13f429006df2464f9026eac27ed51da6 2023-03-27 04:57:27,337 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/668cb90470624e3f85d59ed93aeff70d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/668cb90470624e3f85d59ed93aeff70d 2023-03-27 04:57:27,338 DEBUG [StoreCloser-TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/478be351740b4539a5f38451cde94b1e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/478be351740b4539a5f38451cde94b1e 2023-03-27 04:57:27,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/recovered.edits/432.seqid, newMaxSeqId=432, maxSeqId=1 2023-03-27 04:57:27,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1. 2023-03-27 04:57:27,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for c239f1ca264c8e6332ec629bacfc12b1: 2023-03-27 04:57:27,344 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:27,345 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=c239f1ca264c8e6332ec629bacfc12b1, regionState=CLOSED 2023-03-27 04:57:27,345 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893047345"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679893047345"}]},"ts":"1679893047345"} 2023-03-27 04:57:27,347 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-27 04:57:27,347 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure c239f1ca264c8e6332ec629bacfc12b1, server=jenkins-hbase20.apache.org,44923,1679892955505 in 1.5410 sec 2023-03-27 04:57:27,348 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-27 04:57:27,348 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c239f1ca264c8e6332ec629bacfc12b1, UNASSIGN in 1.5450 sec 2023-03-27 04:57:27,348 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893047348"}]},"ts":"1679893047348"} 2023-03-27 04:57:27,349 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-27 04:57:27,349 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-27 04:57:27,350 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5610 sec 2023-03-27 04:57:27,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-27 04:57:27,901 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-27 04:57:27,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-27 04:57:27,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:27,905 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:27,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-27 04:57:27,906 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:27,910 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:27,914 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/recovered.edits] 2023-03-27 04:57:27,916 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/015ff948151d472cbbe65231689881fa to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/015ff948151d472cbbe65231689881fa 2023-03-27 04:57:27,918 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ed981eecf0d444179f98fe00114f6601 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/A/ed981eecf0d444179f98fe00114f6601 2023-03-27 04:57:27,920 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/2cd0a4b434554d9993f1cd92fd8423a7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/2cd0a4b434554d9993f1cd92fd8423a7 2023-03-27 04:57:27,921 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/8b13269f273b4fc582659958002622ce to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/B/8b13269f273b4fc582659958002622ce 2023-03-27 04:57:27,923 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/4dc0537342864498a8ec39acba53982c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/4dc0537342864498a8ec39acba53982c 2023-03-27 04:57:27,925 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/ddb1bdfb20db4a3b9a471d639bbf2777 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/C/ddb1bdfb20db4a3b9a471d639bbf2777 2023-03-27 04:57:27,929 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/recovered.edits/432.seqid to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1/recovered.edits/432.seqid 2023-03-27 04:57:27,929 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/c239f1ca264c8e6332ec629bacfc12b1 2023-03-27 04:57:27,930 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-27 04:57:27,933 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:27,934 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-27 04:57:27,936 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-27 04:57:27,938 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:27,938 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-27 04:57:27,938 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679893047938"}]},"ts":"9223372036854775807"} 2023-03-27 04:57:27,940 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-27 04:57:27,940 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c239f1ca264c8e6332ec629bacfc12b1, NAME => 'TestAcidGuarantees,,1679893020911.c239f1ca264c8e6332ec629bacfc12b1.', STARTKEY => '', ENDKEY => ''}] 2023-03-27 04:57:27,940 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-27 04:57:27,940 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679893047940"}]},"ts":"9223372036854775807"} 2023-03-27 04:57:27,943 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-27 04:57:27,945 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:27,946 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 43 msec 2023-03-27 04:57:28,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-27 04:57:28,008 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-27 04:57:28,025 INFO [Listener at localhost.localdomain/43311] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=228 (was 227) - Thread LEAK? -, OpenFileDescriptor=559 (was 561), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=463 (was 458) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=849 (was 895) 2023-03-27 04:57:28,034 INFO [Listener at localhost.localdomain/43311] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228, OpenFileDescriptor=559, MaxFileDescriptor=60000, SystemLoadAverage=463, ProcessCount=167, AvailableMemoryMB=849 2023-03-27 04:57:28,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-27 04:57:28,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-27 04:57:28,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:28,039 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-27 04:57:28,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-27 04:57:28,039 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-27 04:57:28,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-27 04:57:28,041 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:28,041 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc empty. 2023-03-27 04:57:28,041 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:28,041 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-27 04:57:28,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-27 04:57:28,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-27 04:57:28,449 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-27 04:57:28,451 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 682db1751885437677f00c4ab8bf8ffc, NAME => 'TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp 2023-03-27 04:57:28,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-27 04:57:28,857 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:57:28,858 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 682db1751885437677f00c4ab8bf8ffc, disabling compactions & flushes 2023-03-27 04:57:28,858 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:28,858 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:28,858 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. after waiting 0 ms 2023-03-27 04:57:28,858 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:28,858 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:28,858 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:28,863 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-27 04:57:28,866 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893048865"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679893048865"}]},"ts":"1679893048865"} 2023-03-27 04:57:28,867 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-27 04:57:28,868 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-27 04:57:28,869 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893048869"}]},"ts":"1679893048869"} 2023-03-27 04:57:28,870 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-27 04:57:28,873 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=682db1751885437677f00c4ab8bf8ffc, ASSIGN}] 2023-03-27 04:57:28,875 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=682db1751885437677f00c4ab8bf8ffc, ASSIGN 2023-03-27 04:57:28,876 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=682db1751885437677f00c4ab8bf8ffc, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,44923,1679892955505; forceNewPlan=false, retain=false 2023-03-27 04:57:29,028 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=682db1751885437677f00c4ab8bf8ffc, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:29,028 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893049028"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679893049028"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679893049028"}]},"ts":"1679893049028"} 2023-03-27 04:57:29,031 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:57:29,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-27 04:57:29,193 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:29,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 682db1751885437677f00c4ab8bf8ffc, NAME => 'TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.', STARTKEY => '', ENDKEY => ''} 2023-03-27 04:57:29,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:29,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:57:29,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:29,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:29,195 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:29,196 DEBUG [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A 2023-03-27 04:57:29,196 DEBUG [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A 2023-03-27 04:57:29,196 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:57:29,197 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 682db1751885437677f00c4ab8bf8ffc columnFamilyName A 2023-03-27 04:57:29,197 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.HStore(310): Store=682db1751885437677f00c4ab8bf8ffc/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:57:29,197 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:29,198 DEBUG [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B 2023-03-27 04:57:29,198 DEBUG [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B 2023-03-27 04:57:29,198 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:57:29,199 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 682db1751885437677f00c4ab8bf8ffc columnFamilyName B 2023-03-27 04:57:29,199 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.HStore(310): Store=682db1751885437677f00c4ab8bf8ffc/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:57:29,199 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:29,200 DEBUG [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C 2023-03-27 04:57:29,200 DEBUG [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C 2023-03-27 04:57:29,200 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:57:29,200 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 682db1751885437677f00c4ab8bf8ffc columnFamilyName C 2023-03-27 04:57:29,201 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.HStore(310): Store=682db1751885437677f00c4ab8bf8ffc/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:57:29,201 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:29,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:29,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:29,203 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-27 04:57:29,204 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:29,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-27 04:57:29,206 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 682db1751885437677f00c4ab8bf8ffc; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61148463, jitterRate=-0.08881689608097076}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-27 04:57:29,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:29,207 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc., pid=37, masterSystemTime=1679893049185 2023-03-27 04:57:29,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:29,208 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:29,209 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=682db1751885437677f00c4ab8bf8ffc, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:29,209 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893049208"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679893049208"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679893049208"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679893049208"}]},"ts":"1679893049208"} 2023-03-27 04:57:29,211 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-27 04:57:29,211 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 in 179 msec 2023-03-27 04:57:29,213 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-27 04:57:29,213 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=682db1751885437677f00c4ab8bf8ffc, ASSIGN in 338 msec 2023-03-27 04:57:29,214 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-27 04:57:29,214 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893049214"}]},"ts":"1679893049214"} 2023-03-27 04:57:29,215 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-27 04:57:29,217 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-27 04:57:29,218 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1800 sec 2023-03-27 04:57:30,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-27 04:57:30,148 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-27 04:57:30,149 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x1bd9eb59 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:30,160 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7889ece0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:30,161 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:30,164 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:32952, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:30,165 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-27 04:57:30,166 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41304, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-27 04:57:30,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-27 04:57:30,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-27 04:57:30,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:30,581 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-27 04:57:30,583 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-27 04:57:30,583 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-27 04:57:30,588 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-27 04:57:30,591 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=682db1751885437677f00c4ab8bf8ffc, REOPEN/MOVE}] 2023-03-27 04:57:30,592 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=682db1751885437677f00c4ab8bf8ffc, REOPEN/MOVE 2023-03-27 04:57:30,592 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=682db1751885437677f00c4ab8bf8ffc, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:30,592 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893050592"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679893050592"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679893050592"}]},"ts":"1679893050592"} 2023-03-27 04:57:30,593 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:57:30,745 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:30,746 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 682db1751885437677f00c4ab8bf8ffc, disabling compactions & flushes 2023-03-27 04:57:30,747 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:30,747 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:30,747 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. after waiting 0 ms 2023-03-27 04:57:30,747 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:30,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-27 04:57:30,753 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:30,753 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:30,753 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 682db1751885437677f00c4ab8bf8ffc to self. 2023-03-27 04:57:30,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:30,756 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=682db1751885437677f00c4ab8bf8ffc, regionState=CLOSED 2023-03-27 04:57:30,756 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893050756"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679893050756"}]},"ts":"1679893050756"} 2023-03-27 04:57:30,760 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-27 04:57:30,760 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 in 165 msec 2023-03-27 04:57:30,761 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=682db1751885437677f00c4ab8bf8ffc, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,44923,1679892955505; forceNewPlan=false, retain=true 2023-03-27 04:57:30,912 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=682db1751885437677f00c4ab8bf8ffc, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:30,912 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893050912"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679893050912"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679893050912"}]},"ts":"1679893050912"} 2023-03-27 04:57:30,915 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:57:31,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 682db1751885437677f00c4ab8bf8ffc, NAME => 'TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.', STARTKEY => '', ENDKEY => ''} 2023-03-27 04:57:31,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:31,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:57:31,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:31,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:31,081 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:31,082 DEBUG [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A 2023-03-27 04:57:31,082 DEBUG [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A 2023-03-27 04:57:31,083 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:57:31,083 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 682db1751885437677f00c4ab8bf8ffc columnFamilyName A 2023-03-27 04:57:31,085 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.HStore(310): Store=682db1751885437677f00c4ab8bf8ffc/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:57:31,085 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:31,086 DEBUG [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B 2023-03-27 04:57:31,086 DEBUG [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B 2023-03-27 04:57:31,086 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:57:31,087 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 682db1751885437677f00c4ab8bf8ffc columnFamilyName B 2023-03-27 04:57:31,087 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.HStore(310): Store=682db1751885437677f00c4ab8bf8ffc/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:57:31,087 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:31,088 DEBUG [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C 2023-03-27 04:57:31,088 DEBUG [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C 2023-03-27 04:57:31,088 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:57:31,089 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 682db1751885437677f00c4ab8bf8ffc columnFamilyName C 2023-03-27 04:57:31,089 INFO [StoreOpener-682db1751885437677f00c4ab8bf8ffc-1] regionserver.HStore(310): Store=682db1751885437677f00c4ab8bf8ffc/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:57:31,089 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:31,091 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:31,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-27 04:57:31,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:31,095 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 682db1751885437677f00c4ab8bf8ffc; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74799983, jitterRate=0.11460660398006439}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-27 04:57:31,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:31,096 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc., pid=42, masterSystemTime=1679893051069 2023-03-27 04:57:31,097 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,097 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,097 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=682db1751885437677f00c4ab8bf8ffc, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,098 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893051097"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679893051097"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679893051097"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679893051097"}]},"ts":"1679893051097"} 2023-03-27 04:57:31,100 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-27 04:57:31,100 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 in 184 msec 2023-03-27 04:57:31,101 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-27 04:57:31,101 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=682db1751885437677f00c4ab8bf8ffc, REOPEN/MOVE in 509 msec 2023-03-27 04:57:31,103 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-27 04:57:31,103 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 514 msec 2023-03-27 04:57:31,105 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 936 msec 2023-03-27 04:57:31,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-27 04:57:31,107 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x7d243e46 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:31,112 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29738198, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:31,112 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x74a355de to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:31,127 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28a98302, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:31,128 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x57b6628e to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:31,135 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39d5c003, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:31,135 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x5da5030e to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:31,142 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39e3a826, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:31,143 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x74d6bf18 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:31,152 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19e7fff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:31,153 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x47e30d1c to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:31,161 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35f0f398, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:31,161 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x36a9ece3 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:31,169 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69fd9cc7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:31,169 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x2ded66ad to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:31,177 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54da4907, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:31,177 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x14d061d2 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:31,185 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52cea5eb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:31,186 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x228ff1a4 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:31,194 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3546dad6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:31,203 DEBUG [hconnection-0x46baf19-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:31,204 DEBUG [hconnection-0x460f8149-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:31,204 DEBUG [hconnection-0x252da29f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:31,204 DEBUG [hconnection-0x667580ad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:31,204 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34566, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:31,204 DEBUG [hconnection-0x7a0e33bf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:31,204 DEBUG [hconnection-0x31ac71dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:31,204 DEBUG [hconnection-0x11aed65a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:31,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:31,204 DEBUG [hconnection-0x463d458c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:31,207 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34588, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:31,207 DEBUG [hconnection-0x6dacff27-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:31,207 DEBUG [hconnection-0x7098cc30-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:31,209 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34586, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:31,209 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34600, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:31,211 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34614, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:31,214 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34572, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:31,214 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34610, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:31,214 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34598, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:31,214 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34628, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:31,214 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34612, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:31,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:31,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:31,218 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:31,218 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:31,218 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:31,220 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:31,220 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,220 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,221 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:31,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:31,221 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:31,221 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,221 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:31,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:31,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:31,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:31,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:31,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:31,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:31,234 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:31,231 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,234 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:31,231 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-27 04:57:31,234 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:31,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:31,234 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,234 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:31,234 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:31,234 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,234 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:31,234 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:31,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:31,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:31,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:31,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:31,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:31,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:31,235 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,235 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,235 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,236 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:31,237 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:31,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6074bf2c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:31,237 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,238 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,238 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:31,238 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:31,238 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:31,238 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,238 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:31,239 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,239 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:31,239 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:31,239 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:31,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:31,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:31,241 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:31,241 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:31,241 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:31,241 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:31,241 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:31,242 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,242 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,243 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:31,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,243 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:31,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893111243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893111243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893111244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,244 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893111244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893111245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,246 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,246 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,246 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:31,246 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:31,246 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:31,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:31,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:31,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:31,246 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:31,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:31,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:31,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:31,246 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,246 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:31,247 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,247 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:31,247 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,247 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893111344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,347 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:31,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893111344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:31,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893111345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893111345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893111345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:31,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:31,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:31,350 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:31,350 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:31,350 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:31,350 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:31,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,351 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:31,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:31,352 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,352 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:31,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:31,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:31,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:31,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:31,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:31,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:31,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:31,352 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,353 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:31,353 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:31,353 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:31,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:31,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:31,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:31,354 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,354 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,355 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,355 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:31,356 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:31,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f989d11[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:31,356 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,356 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,356 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:31,356 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,356 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:31,356 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:31,356 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:31,357 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:31,357 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:31,357 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,357 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:31,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:31,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:31,359 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:31,359 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:31,359 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:31,359 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:31,359 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:31,359 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,360 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,360 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:31,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,360 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:31,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,361 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,362 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,362 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,362 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:31,362 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:31,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:31,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:31,362 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:31,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:31,362 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:31,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:31,362 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,363 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,363 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,363 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,363 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:31,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:31,363 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:31,463 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:31,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:31,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:31,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:31,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:31,468 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:31,468 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:31,468 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:31,468 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:31,469 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,469 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,469 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:31,469 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:31,470 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,470 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:31,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:31,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:31,474 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:31,474 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:31,474 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:31,474 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:31,474 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:31,474 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,474 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:31,474 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:31,474 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,474 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:31,474 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:31,474 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,474 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:31,475 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:31,483 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,483 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,484 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,484 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:31,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d846bfa[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:31,485 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:31,485 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,485 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,485 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:31,485 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,485 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:31,485 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:31,485 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:31,486 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,486 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,486 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:31,486 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:31,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:31,488 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:31,488 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:31,488 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:31,488 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:31,488 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:31,489 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,489 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:31,489 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,489 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,490 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:31,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:31,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,492 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:31,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,492 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,492 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,492 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,492 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,492 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,492 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,492 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:31,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:31,492 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:31,492 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:31,492 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:31,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:31,492 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:31,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:31,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:31,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:31,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:31,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:31,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893111548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893111548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893111548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893111550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893111550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,592 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:31,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:31,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:31,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:31,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:31,595 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:31,595 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:31,595 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:31,595 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:31,596 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,596 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,596 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,596 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:31,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:31,596 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,596 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:31,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:31,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:31,597 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:31,597 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:31,597 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:31,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:31,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:31,598 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,598 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:31,598 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:31,598 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:31,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:31,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:31,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:31,599 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,599 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,599 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,599 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:31,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68930796[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:31,600 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:31,600 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,600 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:31,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:31,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:31,601 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,601 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:31,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:31,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:31,601 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:31,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:31,603 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:31,603 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:31,603 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:31,603 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:31,603 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:31,604 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,604 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,604 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,604 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,605 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:31,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:31,607 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:31,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:31,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:31,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:31,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:31,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:31,607 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:31,607 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:31,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:31,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:31,672 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303276c3d11b4b5a34996b8eca1b417d52f9d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303276c3d11b4b5a34996b8eca1b417d52f9d 2023-03-27 04:57:31,707 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:31,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:31,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:31,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:31,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:31,710 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:31,710 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:31,710 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:31,710 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:31,710 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,710 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:31,711 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:31,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:31,711 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,711 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:31,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:31,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:31,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:31,712 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:31,712 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:31,712 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:31,713 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,713 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:31,713 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:31,713 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:31,714 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,714 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,714 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,714 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:31,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ddb5877[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:31,716 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:31,716 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,717 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,717 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:31,717 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:31,717 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,717 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:31,717 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:31,718 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,718 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:31,718 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,718 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:31,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:31,720 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:31,720 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:31,720 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:31,720 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:31,720 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:31,720 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,720 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,720 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:31,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:31,721 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:31,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:31,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:31,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,725 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:31,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:31,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:31,725 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:31,725 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:31,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:31,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:31,825 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:31,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:31,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:31,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:31,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:31,828 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:31,828 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:31,828 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:31,828 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:31,829 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,829 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,829 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:31,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:31,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:31,829 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,830 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:31,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:31,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:31,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:31,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:31,830 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:31,830 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:31,830 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:31,830 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:31,831 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,831 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:31,831 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:31,831 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:31,832 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,832 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,832 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,832 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:31,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@169028c8[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:31,833 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:31,833 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,834 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,834 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:31,834 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:31,834 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:31,834 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,834 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:31,835 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,835 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:31,835 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,835 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:31,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:31,838 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:31,838 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:31,838 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:31,838 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:31,838 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:31,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:31,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,839 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:31,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:31,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,840 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,841 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:31,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:31,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:31,842 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:31,842 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,842 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:31,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:31,842 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:31,842 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:31,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:31,842 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:31,842 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:31,842 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:31,842 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,842 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,842 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893111852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893111853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893111854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893111855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:31,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893111856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,942 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:31,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:31,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:31,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:31,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:31,945 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:31,945 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:31,945 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:31,945 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:31,945 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,945 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,946 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,946 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:31,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:31,946 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,946 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:31,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:31,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:31,947 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:31,947 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:31,947 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:31,947 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:31,947 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:31,947 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,947 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:31,947 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:31,947 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:31,948 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:31,948 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:31,948 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,948 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:31,948 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:31,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,949 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,949 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,949 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:31,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:31,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68355a5b[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:31,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,951 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:31,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:31,951 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,951 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:31,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:31,951 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:31,952 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,952 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:31,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:31,953 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:31,954 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:31,954 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:31,954 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:31,954 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:31,954 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,954 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,954 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:31,955 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:31,956 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:31,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:31,958 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:31,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:31,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:31,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:31,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:31,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:31,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:31,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:31,958 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:31,958 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:31,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:31,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:32,058 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:32,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:32,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:32,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:32,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:32,062 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:32,062 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:32,062 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:32,062 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:32,062 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,062 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,063 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:32,063 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:32,063 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,063 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:32,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:32,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:32,064 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:32,064 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:32,064 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:32,064 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:32,064 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:32,064 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:32,064 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:32,064 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:32,064 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:32,064 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:32,064 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:32,064 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,065 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:32,065 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:32,065 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,065 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,066 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,066 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:32,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@186a5f7c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:32,068 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:32,068 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,069 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,069 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:32,069 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,069 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:32,069 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:32,069 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:32,070 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,070 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,070 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:32,070 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:32,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:32,074 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:32,074 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:32,074 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:32,074 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:32,074 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:32,075 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,075 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,075 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:32,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:32,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:32,075 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,076 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,077 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=21, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/50bab4d4ff8e49eba9ffa453f9e25522 2023-03-27 04:57:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,079 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:32,079 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:32,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:32,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:32,083 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,083 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,083 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,083 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:32,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:32,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:32,083 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,083 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,084 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,084 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,084 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:32,084 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:32,184 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:32,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:32,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:32,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:32,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:32,187 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:32,187 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:32,187 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:32,187 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:32,187 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,187 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,188 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,188 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:32,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:32,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:32,188 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,188 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:32,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:32,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:32,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:32,189 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:32,189 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:32,189 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:32,189 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:32,189 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:32,189 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:32,189 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:32,189 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:32,189 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:32,189 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:32,189 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:32,189 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,190 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:32,190 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:32,190 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,190 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,191 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,191 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:32,192 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:32,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f05a4e6[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:32,192 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,192 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,192 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:32,192 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,192 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:32,192 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:32,192 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:32,193 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,193 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:32,193 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:32,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,193 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:32,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:32,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:32,195 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:32,195 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:32,195 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:32,195 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:32,195 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:32,195 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,195 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,195 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:32,196 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,197 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:32,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:32,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:32,198 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:32,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:32,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:32,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,198 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:32,198 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:32,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:32,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:32,298 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:32,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:32,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:32,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:32,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:32,301 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:32,301 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:32,301 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:32,301 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:32,302 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,302 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,302 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:32,302 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:32,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:32,303 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,303 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:32,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:32,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:32,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:32,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:32,304 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:32,304 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:32,304 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:32,304 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:32,305 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,305 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,305 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,305 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:32,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c417a6[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:32,306 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:32,306 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,306 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,307 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:32,307 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:32,307 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:32,307 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,307 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:32,307 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,307 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,307 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:32,307 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:32,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:32,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:32,309 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:32,309 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:32,309 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:32,309 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:32,309 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:32,310 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,310 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,310 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:32,310 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,311 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,312 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,312 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,312 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,312 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:32,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:32,312 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:32,312 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,312 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:32,312 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:32,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:32,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:32,312 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:32,313 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:32,313 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,313 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:32,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:32,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:32,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:32,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893112365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:32,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893112365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:32,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893112365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:32,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893112365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:32,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893112365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,413 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:32,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:32,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:32,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:32,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:32,415 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:32,416 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:32,416 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:32,416 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:32,416 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,416 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,416 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,416 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:32,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:32,417 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,417 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:32,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:32,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:32,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:32,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:32,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:32,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:32,418 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:32,418 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:32,418 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:32,418 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:32,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:32,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:32,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:32,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:32,419 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,419 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,419 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,420 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:32,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:32,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a617d67[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:32,422 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:32,422 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,423 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,423 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:32,423 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:32,423 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,423 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:32,423 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:32,423 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,423 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,423 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:32,424 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:32,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:32,427 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:32,427 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:32,427 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:32,427 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:32,427 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:32,427 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,427 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,428 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:32,428 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,431 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,433 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:32,433 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:32,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:32,433 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,433 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,433 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,433 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,433 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,433 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,433 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:32,433 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:32,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:32,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:32,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:32,433 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:32,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:32,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:32,433 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:32,500 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=21 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/4a9bfa1a0d4549aab799356053fedff5 2023-03-27 04:57:32,534 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:32,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:32,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:32,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:32,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:32,536 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:32,536 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:32,536 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:32,536 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:32,537 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,537 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,537 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:32,537 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:32,538 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,538 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:32,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:32,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:32,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:32,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:32,538 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:32,538 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:32,539 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:32,539 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:32,539 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:32,539 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:32,539 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:32,539 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:32,539 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:32,539 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:32,539 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:32,539 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,539 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:32,539 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:32,540 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,540 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,540 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,540 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:32,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ba54e8[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:32,542 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:32,542 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,542 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,542 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:32,542 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,542 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:32,542 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:32,542 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:32,543 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,543 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:32,543 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:32,543 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:32,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:32,546 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:32,546 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:32,546 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:32,546 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:32,546 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:32,547 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,547 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,547 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:32,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:32,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:32,548 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,549 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,550 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,550 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,550 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,550 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,550 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,550 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:32,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:32,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:32,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:32,550 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:32,550 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:32,550 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:32,550 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:32,550 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,550 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:32,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:32,651 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:32,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:32,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:32,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:32,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:32,653 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:32,653 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:32,653 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:32,653 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:32,654 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,654 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,654 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,654 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:32,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:32,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:32,654 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,654 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:32,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:32,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:32,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:32,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:32,655 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:32,655 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:32,655 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:32,656 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:32,656 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:32,656 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:32,656 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:32,656 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:32,656 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:32,656 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:32,656 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:32,656 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,656 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:32,656 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:32,657 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,657 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,657 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,657 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:32,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:32,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32daec8d[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:32,658 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:32,658 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,659 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:32,659 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,659 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:32,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:32,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:32,660 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,660 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:32,660 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:32,660 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:32,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:32,663 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:32,663 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:32,663 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:32,663 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:32,663 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:32,664 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,664 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,664 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:32,664 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,665 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,667 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:32,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:32,667 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:32,667 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:32,667 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:32,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:32,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:32,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:32,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:32,667 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,667 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,667 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,667 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:32,667 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:32,667 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,667 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:32,667 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,767 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:32,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:32,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:32,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:32,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:32,770 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:32,770 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:32,770 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:32,770 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:32,770 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,770 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,771 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,771 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:32,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:32,771 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,771 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:32,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:32,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:32,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:32,772 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:32,772 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:32,772 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:32,772 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:32,772 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:32,772 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:32,772 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:32,772 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:32,772 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:32,772 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:32,772 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:32,772 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,772 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:32,772 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:32,773 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,773 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,773 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,774 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:32,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:32,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6afbb457[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:32,775 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:32,775 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,775 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,775 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:32,775 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:32,775 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,775 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:32,775 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:32,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:32,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:32,776 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:32,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:32,778 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:32,778 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:32,778 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:32,778 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:32,778 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:32,779 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,779 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:32,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,779 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:32,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:32,779 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,780 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:32,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:32,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:32,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:32,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,782 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:32,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:32,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:32,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:32,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:32,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:32,783 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:32,783 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:32,883 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:32,883 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-27 04:57:32,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:32,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:32,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:32,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:32,885 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:32,885 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:32,885 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:32,885 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:32,887 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,887 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,888 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:32,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:32,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,891 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,891 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,891 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:32,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:32,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:32,893 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:32,893 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:32,893 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:32,894 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:32,894 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:32,896 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:32,896 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:32,896 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:32,896 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:32,896 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:32,896 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:32,896 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,897 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:32,897 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:32,910 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,910 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,911 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=21 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/ba80f5ccfad2408b97bec519458ace2c 2023-03-27 04:57:32,911 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,911 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:32,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b40762a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:32,914 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:32,914 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,914 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/50bab4d4ff8e49eba9ffa453f9e25522 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/50bab4d4ff8e49eba9ffa453f9e25522 2023-03-27 04:57:32,917 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,917 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/50bab4d4ff8e49eba9ffa453f9e25522, entries=150, sequenceid=21, filesize=25.1 K 2023-03-27 04:57:32,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,917 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,917 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:32,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,917 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:32,918 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:32,918 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:32,918 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/4a9bfa1a0d4549aab799356053fedff5 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/4a9bfa1a0d4549aab799356053fedff5 2023-03-27 04:57:32,918 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:32,918 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:32,918 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:32,918 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:32,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:32,921 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:32,921 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:32,921 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:32,921 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:32,921 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:32,921 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/4a9bfa1a0d4549aab799356053fedff5, entries=150, sequenceid=21, filesize=11.6 K 2023-03-27 04:57:32,922 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/ba80f5ccfad2408b97bec519458ace2c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/ba80f5ccfad2408b97bec519458ace2c 2023-03-27 04:57:32,922 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,922 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:32,922 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:32,922 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:32,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:32,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:32,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:32,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,924 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,925 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/ba80f5ccfad2408b97bec519458ace2c, entries=150, sequenceid=21, filesize=11.6 K 2023-03-27 04:57:32,926 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=93.93 KB/96180 for 682db1751885437677f00c4ab8bf8ffc in 1695ms, sequenceid=21, compaction requested=false 2023-03-27 04:57:32,926 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:32,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,937 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,937 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:32,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:32,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:32,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:32,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:32,937 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:32,937 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:32,937 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,937 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:32,937 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:32,937 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:32,937 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:32,938 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:32,938 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:32,938 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,038 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:33,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:33,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:33,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:33,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:33,041 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:33,041 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:33,041 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:33,041 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:33,041 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,041 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,042 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:33,042 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:33,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:33,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,042 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,042 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:33,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:33,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:33,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:33,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:33,043 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:33,043 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:33,043 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:33,043 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:33,043 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:33,043 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:33,043 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:33,043 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-27 04:57:33,044 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:33,044 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:33,044 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:33,044 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:33,044 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:33,044 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:33,053 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032716bf1603d7ff43428756b14a4e8fae40 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032716bf1603d7ff43428756b14a4e8fae40 2023-03-27 04:57:33,058 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=38, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/76eb603b9a8e4b4791dcf4760e99b78f 2023-03-27 04:57:33,081 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/882b9ea524e74d24862538ebeaf79cc8 2023-03-27 04:57:33,103 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/3505c96dc9474b0c8d36e0524be9a133 2023-03-27 04:57:33,106 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/76eb603b9a8e4b4791dcf4760e99b78f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/76eb603b9a8e4b4791dcf4760e99b78f 2023-03-27 04:57:33,112 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/76eb603b9a8e4b4791dcf4760e99b78f, entries=150, sequenceid=38, filesize=25.1 K 2023-03-27 04:57:33,113 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/882b9ea524e74d24862538ebeaf79cc8 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/882b9ea524e74d24862538ebeaf79cc8 2023-03-27 04:57:33,118 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/882b9ea524e74d24862538ebeaf79cc8, entries=150, sequenceid=38, filesize=11.6 K 2023-03-27 04:57:33,119 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/3505c96dc9474b0c8d36e0524be9a133 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/3505c96dc9474b0c8d36e0524be9a133 2023-03-27 04:57:33,122 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/3505c96dc9474b0c8d36e0524be9a133, entries=150, sequenceid=38, filesize=11.6 K 2023-03-27 04:57:33,123 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 682db1751885437677f00c4ab8bf8ffc in 80ms, sequenceid=38, compaction requested=false 2023-03-27 04:57:33,123 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:33,123 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:33,123 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:33,123 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:33,123 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:33,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:33,128 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,128 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,128 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,128 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:33,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79ff74be[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:33,129 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:33,129 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,130 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,130 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:33,130 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:33,130 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:33,130 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,130 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:33,131 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:33,131 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:33,131 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:33,134 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:33,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:33,136 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:33,136 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:33,136 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:33,136 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:33,136 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:33,142 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,142 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:33,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,142 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:33,142 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,143 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,151 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:33,151 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:33,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:33,151 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,151 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,151 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,151 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,152 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:33,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:33,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:33,151 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:33,151 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:33,152 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:33,152 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,152 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:33,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:33,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:33,252 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:33,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:33,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:33,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:33,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:33,256 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:33,256 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:33,256 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:33,256 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:33,256 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,256 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,257 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:33,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:33,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:33,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:33,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:33,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:33,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:33,258 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:33,258 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:33,258 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:33,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:33,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:33,259 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:33,259 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:33,259 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:33,259 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:33,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:33,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:33,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:33,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:33,260 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,260 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,260 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,260 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:33,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:33,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ca45188[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:33,262 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:33,262 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,275 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,275 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:33,275 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,275 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:33,275 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:33,275 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:33,276 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:33,276 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:33,276 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:33,276 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:33,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:33,279 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:33,279 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:33,279 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:33,279 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:33,279 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:33,280 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,280 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,280 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:33,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,281 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:33,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:33,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,282 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,284 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:33,284 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:33,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:33,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:33,284 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,284 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,284 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,284 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,284 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:33,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:33,284 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:33,284 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,285 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:33,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:33,285 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:33,285 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:33,385 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:33,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:33,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:33,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:33,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:33,388 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:33,388 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:33,388 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:33,388 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:33,388 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,388 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,389 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:33,389 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:33,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:33,389 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,389 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:33,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:33,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:33,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:33,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:33,390 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:33,390 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:33,390 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:33,391 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:33,391 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:33,391 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:33,391 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:33,391 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:33,391 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:33,391 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:33,391 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:33,391 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,392 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:33,392 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:33,392 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,392 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:33,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:33,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b4a0747[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:33,394 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:33,394 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,395 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,395 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:33,395 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:33,395 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,395 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:33,395 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:33,396 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,396 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:33,396 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:33,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,396 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:33,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:33,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:33,398 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:33,398 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:33,398 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:33,398 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:33,398 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:33,399 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,400 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,400 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:33,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,400 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,401 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,403 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:33,403 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:33,403 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,403 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,403 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:33,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:33,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:33,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:33,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:33,403 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,404 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,404 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,404 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,404 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:33,404 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:33,404 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:33,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:33,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:33,460 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:57:33,460 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:33,460 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:33,460 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:33,460 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:33,460 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:33,460 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:33,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679893053481, max timestamp=1679893053489Num uniques 859; 2023-03-27 04:57:33,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-27 04:57:33,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679893053481, max timestamp=1679893053489Num uniques 859; 2023-03-27 04:57:33,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-27 04:57:33,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679893053481, max timestamp=1679893053489Num uniques 859; 2023-03-27 04:57:33,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-27 04:57:33,504 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:33,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:33,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:33,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:33,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:33,507 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:33,507 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:33,507 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:33,507 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:33,508 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,508 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,508 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:33,508 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:33,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:33,509 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,509 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:33,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:33,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:33,510 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:33,510 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:33,510 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:33,510 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:33,510 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:33,510 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:33,510 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:33,510 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:33,510 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:33,510 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:33,510 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:33,510 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,510 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:33,511 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:33,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,512 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,512 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,512 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:33,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73e3c522[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:33,514 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:33,514 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,514 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,514 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:33,514 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:33,515 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:33,515 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,515 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:33,515 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,515 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:33,515 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:33,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,515 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:33,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:33,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:33,517 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:33,517 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:33,517 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:33,517 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:33,517 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:33,519 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:33,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893113510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,519 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:33,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893113511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,519 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:33,519 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:33,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893113513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:33,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:33,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,521 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,522 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,522 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,522 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:33,523 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:33,523 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:33,523 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,523 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:33,523 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:33,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:33,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:33,523 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,523 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:33,523 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:33,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:33,523 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:33,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:33,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893113519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:33,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893113519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,623 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:33,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:33,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:33,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893113620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:33,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893113620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:33,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893113620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:33,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:33,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:33,626 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:33,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:33,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:33,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:33,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,627 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:33,627 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:33,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:33,627 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,627 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:33,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:33,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:33,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:33,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:33,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:33,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:33,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:33,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:33,628 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:33,628 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:33,628 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:33,628 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:33,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:33,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:33,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:33,629 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:33,629 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,629 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,629 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:33,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71bb0564[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:33,631 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:33,631 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,631 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,631 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:33,632 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,632 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:33,632 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:33,632 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:33,633 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,633 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,633 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:33,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:33,633 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:33,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893113630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:33,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893113631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:33,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:33,635 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:33,635 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:33,635 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:33,635 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:33,635 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:33,636 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,636 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,636 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:33,636 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,637 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,638 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,638 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:33,638 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:33,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:33,639 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,639 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,639 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,639 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,639 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,638 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:33,639 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:33,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:33,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:33,639 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:33,640 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:33,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:33,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:33,740 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:33,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:33,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:33,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:33,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:33,742 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:33,742 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:33,742 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:33,742 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:33,742 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,742 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,743 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,743 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:33,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:33,743 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,743 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:33,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:33,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:33,744 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:33,744 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:33,744 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:33,744 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:33,744 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:33,744 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:33,744 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:33,745 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:33,745 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:33,745 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:33,745 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:33,745 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,745 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:33,745 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:33,745 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,745 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,746 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,746 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:33,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:33,747 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:33,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fad22ca[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:33,747 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,748 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,748 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:33,748 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,749 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:33,749 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:33,749 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:33,752 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:33,752 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,752 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,752 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:33,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:33,754 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:33,755 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:33,755 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:33,755 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:33,755 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:33,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,756 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:33,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,756 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:33,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:33,756 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,757 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,758 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,758 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,758 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,758 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:33,758 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:33,758 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:33,758 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:33,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:33,759 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:33,759 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:33,759 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:33,759 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:33,759 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:33,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:33,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893113825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:33,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893113825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:33,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893113826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,840 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:33,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893113835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,840 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:33,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893113835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,859 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:33,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:33,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:33,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:33,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:33,862 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:33,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:33,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:33,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:33,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,863 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:33,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:33,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:33,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:33,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:33,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:33,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:33,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:33,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:33,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:33,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:33,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:33,864 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:33,864 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:33,864 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:33,864 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:33,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:33,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:33,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:33,864 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:33,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,865 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:33,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:33,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34267447[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:33,866 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:33,866 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,867 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,867 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:33,867 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,867 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:33,867 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:33,867 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:33,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:33,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:33,868 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:33,868 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:33,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:33,870 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:33,870 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:33,870 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:33,870 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:33,870 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:33,870 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,870 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,870 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:33,871 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:33,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:33,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,874 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032765fc3a4a9281465bb4f102f3fc2ebe6d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032765fc3a4a9281465bb4f102f3fc2ebe6d 2023-03-27 04:57:33,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:33,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:33,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:33,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,875 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:33,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:33,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:33,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:33,876 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:33,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:33,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:33,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:33,876 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:33,976 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:33,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:33,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:33,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:33,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:33,978 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:33,978 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:33,978 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:33,978 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:33,979 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,979 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,980 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:33,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:33,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:33,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:33,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:33,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:33,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:33,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:33,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:33,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:33,981 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:33,981 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:33,981 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:33,981 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:33,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:33,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:33,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:33,981 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:33,982 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,982 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,982 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,982 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:33,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:33,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59016d34[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:33,983 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:33,983 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,984 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:33,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:33,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:33,985 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,985 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:33,985 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,985 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:33,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,985 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:33,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:33,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:33,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:33,988 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:33,988 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:33,988 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:33,988 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:33,988 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:33,989 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,989 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:33,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,989 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:33,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:33,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:33,991 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:33,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:33,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,992 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:33,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:33,992 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:33,992 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:33,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:33,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:33,993 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,993 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:33,993 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:33,993 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:33,993 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:33,993 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:33,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:33,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:33,993 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:33,993 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:33,993 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,093 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:34,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:34,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:34,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:34,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:34,096 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:34,096 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:34,096 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:34,096 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:34,096 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,096 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,097 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:34,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:34,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:34,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,097 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,098 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,098 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:34,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:34,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:34,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:34,098 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:34,098 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:34,098 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:34,098 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:34,098 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:34,098 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:34,098 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:34,099 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:34,099 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:34,099 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:34,099 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:34,099 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,099 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:34,099 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:34,099 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:34,100 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,103 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,103 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:34,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:34,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70b8c355[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:34,104 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:34,104 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,108 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,109 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,109 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:34,109 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:34,109 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:34,109 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:34,110 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:34,110 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,110 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:34,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:34,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:34,110 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:34,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:34,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:34,112 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:34,112 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:34,112 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:34,112 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:34,112 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:34,113 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,113 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:34,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,113 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:34,113 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:34,115 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:34,117 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:34,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:34,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:34,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:34,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:34,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:34,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,117 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:34,117 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:34,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:34,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:34,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:34,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:34,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893114130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:34,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893114130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:34,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893114130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:34,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893114142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:34,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893114143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,217 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:34,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:34,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:34,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:34,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:34,220 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:34,220 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:34,220 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:34,220 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:34,220 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,220 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,221 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,221 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:34,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:34,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:34,221 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,221 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:34,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:34,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:34,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:34,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:34,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:34,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:34,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:34,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:34,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:34,222 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:34,222 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:34,222 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:34,222 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:34,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:34,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:34,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:34,222 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:34,223 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,223 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:34,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:34,223 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,223 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:34,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:34,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:34,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56a514fa[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:34,224 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:34,224 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,225 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,225 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:34,225 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:34,225 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,225 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:34,225 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:34,226 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:34,226 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:34,226 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:34,226 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:34,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:34,227 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:34,227 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:34,227 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:34,227 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:34,227 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:34,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,228 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:34,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,229 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,230 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,231 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,231 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:34,231 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:34,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:34,231 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:34,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:34,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:34,231 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,231 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:34,231 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:34,231 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:34,231 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,231 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,231 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,231 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,278 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/f801182b8e4b4ca9b33b4181ea6a91df 2023-03-27 04:57:34,285 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/f614555e35d94808bd369bdc776f62a1 2023-03-27 04:57:34,331 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:34,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:34,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:34,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:34,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:34,334 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:34,334 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:34,334 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:34,334 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:34,334 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,334 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,335 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,335 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:34,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:34,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:34,335 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,335 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:34,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:34,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:34,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:34,336 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:34,336 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:34,336 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:34,336 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:34,336 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:34,336 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:34,336 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:34,336 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:34,337 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:34,337 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:34,337 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:34,337 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,337 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:34,337 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:34,337 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,337 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:34,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:34,338 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,338 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:34,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:34,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:34,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:34,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:34,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5126d625[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:34,340 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:34,340 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,341 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,341 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,341 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:34,341 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:34,341 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:34,341 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:34,342 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,342 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:34,342 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:34,342 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:34,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:34,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:34,344 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:34,344 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:34,344 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:34,344 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:34,344 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:34,345 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,345 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,345 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:34,345 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,346 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,347 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,348 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:34,348 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,348 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,348 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,348 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,348 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,348 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:34,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:34,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:34,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:34,348 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:34,348 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:34,348 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:34,348 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:34,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:34,448 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:34,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:34,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:34,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:34,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:34,451 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:34,451 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:34,451 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:34,451 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:34,451 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,451 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,452 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,452 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:34,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:34,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:34,452 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,452 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:34,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:34,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:34,453 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:34,453 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:34,453 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:34,453 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:34,453 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:34,453 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:34,454 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:34,454 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:34,454 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:34,454 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:34,454 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:34,454 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,454 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:34,454 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:34,455 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,455 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:34,455 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,455 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:34,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:34,456 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:34,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39c79f8[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:34,456 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,456 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,456 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,456 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:34,456 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:34,457 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:34,457 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:34,457 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:34,457 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:34,457 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:34,457 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:34,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:34,459 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:34,459 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:34,459 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:34,459 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:34,459 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:34,460 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,460 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:34,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:34,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,460 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:34,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:34,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:34,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:34,461 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,462 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,464 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,464 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:34,464 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:34,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:34,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:34,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:34,464 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:34,464 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:34,464 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,464 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:34,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:34,464 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:34,464 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,464 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:34,464 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,464 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,564 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:34,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:34,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:34,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:34,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:34,567 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:34,567 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:34,567 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:34,567 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:34,568 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,568 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,576 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,576 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:34,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:34,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:34,577 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,577 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:34,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:34,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:34,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:34,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:34,577 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:34,577 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:34,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:34,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:34,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:34,578 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:34,578 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:34,578 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:34,578 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:34,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:34,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:34,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:34,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:34,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:34,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893114633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:34,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893114634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:34,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893114635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,640 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,640 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:34,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:34,640 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:34,640 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:34,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:34,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:34,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3508a0b5[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:34,641 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:34,641 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,642 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,642 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:34,642 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:34,642 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:34,642 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,642 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:34,643 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,643 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:34,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:34,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:34,643 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:34,643 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:34,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:34,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:34,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:34,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:34,645 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:34,645 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:34,645 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:34,645 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:34,645 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:34,646 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,646 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,646 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:34,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:34,647 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,648 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,650 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,650 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,650 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:34,650 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:34,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:34,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:34,650 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,650 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:34,650 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:34,650 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:34,650 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:34,650 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:34,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:34,650 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:34,650 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:34,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893114652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:34,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893114652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,693 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/8496954652f34711a9e69c67e95accd3 2023-03-27 04:57:34,697 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/f801182b8e4b4ca9b33b4181ea6a91df as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/f801182b8e4b4ca9b33b4181ea6a91df 2023-03-27 04:57:34,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/f801182b8e4b4ca9b33b4181ea6a91df, entries=200, sequenceid=50, filesize=31.9 K 2023-03-27 04:57:34,701 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/f614555e35d94808bd369bdc776f62a1 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/f614555e35d94808bd369bdc776f62a1 2023-03-27 04:57:34,705 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/f614555e35d94808bd369bdc776f62a1, entries=150, sequenceid=50, filesize=11.6 K 2023-03-27 04:57:34,705 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/8496954652f34711a9e69c67e95accd3 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8496954652f34711a9e69c67e95accd3 2023-03-27 04:57:34,710 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8496954652f34711a9e69c67e95accd3, entries=150, sequenceid=50, filesize=11.6 K 2023-03-27 04:57:34,711 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 682db1751885437677f00c4ab8bf8ffc in 1251ms, sequenceid=50, compaction requested=true 2023-03-27 04:57:34,711 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:34,711 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:34,711 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:34,711 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:34,711 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:34,711 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:34,712 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:34,712 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:34,712 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 682db1751885437677f00c4ab8bf8ffc/B is initiating minor compaction (all files) 2023-03-27 04:57:34,712 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 682db1751885437677f00c4ab8bf8ffc/A is initiating minor compaction (all files) 2023-03-27 04:57:34,712 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 682db1751885437677f00c4ab8bf8ffc/B in TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:34,712 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 682db1751885437677f00c4ab8bf8ffc/A in TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:34,712 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/4a9bfa1a0d4549aab799356053fedff5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/882b9ea524e74d24862538ebeaf79cc8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/f614555e35d94808bd369bdc776f62a1] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp, totalSize=34.8 K 2023-03-27 04:57:34,712 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/50bab4d4ff8e49eba9ffa453f9e25522, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/76eb603b9a8e4b4791dcf4760e99b78f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/f801182b8e4b4ca9b33b4181ea6a91df] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp, totalSize=82.2 K 2023-03-27 04:57:34,713 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 4a9bfa1a0d4549aab799356053fedff5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=21, earliestPutTs=1679893051227 2023-03-27 04:57:34,713 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 50bab4d4ff8e49eba9ffa453f9e25522, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=21, earliestPutTs=1679893051227 2023-03-27 04:57:34,713 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 882b9ea524e74d24862538ebeaf79cc8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1679893051240 2023-03-27 04:57:34,713 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 76eb603b9a8e4b4791dcf4760e99b78f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1679893051240 2023-03-27 04:57:34,713 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting f614555e35d94808bd369bdc776f62a1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679893053424 2023-03-27 04:57:34,713 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting f801182b8e4b4ca9b33b4181ea6a91df, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679893053423 2023-03-27 04:57:34,723 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 682db1751885437677f00c4ab8bf8ffc#B#compaction#307 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:34,728 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 682db1751885437677f00c4ab8bf8ffc#A#compaction#308 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:34,736 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/3022553b40534628aa5ff28a957ce511 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/3022553b40534628aa5ff28a957ce511 2023-03-27 04:57:34,739 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 682db1751885437677f00c4ab8bf8ffc/B of 682db1751885437677f00c4ab8bf8ffc into 3022553b40534628aa5ff28a957ce511(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:34,739 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:34,739 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc., storeName=682db1751885437677f00c4ab8bf8ffc/B, priority=13, startTime=1679893054711; duration=0sec 2023-03-27 04:57:34,739 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:34,739 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:57:34,740 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:57:34,740 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 682db1751885437677f00c4ab8bf8ffc/C is initiating minor compaction (all files) 2023-03-27 04:57:34,740 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 682db1751885437677f00c4ab8bf8ffc/C in TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:34,740 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/ba80f5ccfad2408b97bec519458ace2c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/3505c96dc9474b0c8d36e0524be9a133, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8496954652f34711a9e69c67e95accd3] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp, totalSize=34.8 K 2023-03-27 04:57:34,741 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting ba80f5ccfad2408b97bec519458ace2c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=21, earliestPutTs=1679893051227 2023-03-27 04:57:34,741 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 3505c96dc9474b0c8d36e0524be9a133, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1679893051240 2023-03-27 04:57:34,741 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 8496954652f34711a9e69c67e95accd3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679893053424 2023-03-27 04:57:34,746 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 682db1751885437677f00c4ab8bf8ffc#C#compaction#309 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:34,750 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:34,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:34,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:34,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:34,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:34,754 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:34,754 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:34,754 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:34,754 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:34,754 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,754 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:34,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:34,755 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,755 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:34,755 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:34,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:34,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:34,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:34,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:34,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:34,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:34,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:34,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:34,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:34,756 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:34,756 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:34,756 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-03-27 04:57:34,757 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:34,757 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:34,757 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:34,757 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:34,757 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:34,757 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:35,147 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/218903ba93c84d238678c09ada4bdf47 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/218903ba93c84d238678c09ada4bdf47 2023-03-27 04:57:35,150 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 682db1751885437677f00c4ab8bf8ffc/A of 682db1751885437677f00c4ab8bf8ffc into 218903ba93c84d238678c09ada4bdf47(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:35,151 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:35,151 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc., storeName=682db1751885437677f00c4ab8bf8ffc/A, priority=13, startTime=1679893054711; duration=0sec 2023-03-27 04:57:35,151 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:35,153 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/8c575df33b18447ea1aba69b8c7e287c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8c575df33b18447ea1aba69b8c7e287c 2023-03-27 04:57:35,156 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 682db1751885437677f00c4ab8bf8ffc/C of 682db1751885437677f00c4ab8bf8ffc into 8c575df33b18447ea1aba69b8c7e287c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:35,156 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:35,156 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc., storeName=682db1751885437677f00c4ab8bf8ffc/C, priority=13, startTime=1679893054711; duration=0sec 2023-03-27 04:57:35,156 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:35,173 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327e2f63be8409841a883e88a6aceb01a51 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e2f63be8409841a883e88a6aceb01a51 2023-03-27 04:57:35,577 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=87, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/f8b64f2bbe704392b8ba5182f11c03ff 2023-03-27 04:57:35,586 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/faea3cab2a64481592b10dca3fe8de41 2023-03-27 04:57:35,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:35,640 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:35,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:35,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893115648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:35,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:35,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893115648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:35,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:35,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893115650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:35,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:35,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893115663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:35,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:35,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893115663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:35,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:35,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893115751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:35,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:35,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893115754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:35,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:35,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893115755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:35,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:35,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893115957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:35,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:35,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893115957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:35,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:35,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893115959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,000 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/2ef90a4bbdc747aaa51a7351d23ab60d 2023-03-27 04:57:36,003 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/f8b64f2bbe704392b8ba5182f11c03ff as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/f8b64f2bbe704392b8ba5182f11c03ff 2023-03-27 04:57:36,007 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/f8b64f2bbe704392b8ba5182f11c03ff, entries=150, sequenceid=87, filesize=25.1 K 2023-03-27 04:57:36,007 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/faea3cab2a64481592b10dca3fe8de41 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/faea3cab2a64481592b10dca3fe8de41 2023-03-27 04:57:36,010 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/faea3cab2a64481592b10dca3fe8de41, entries=150, sequenceid=87, filesize=11.6 K 2023-03-27 04:57:36,011 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/2ef90a4bbdc747aaa51a7351d23ab60d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/2ef90a4bbdc747aaa51a7351d23ab60d 2023-03-27 04:57:36,018 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/2ef90a4bbdc747aaa51a7351d23ab60d, entries=150, sequenceid=87, filesize=11.6 K 2023-03-27 04:57:36,019 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 682db1751885437677f00c4ab8bf8ffc in 1263ms, sequenceid=87, compaction requested=false 2023-03-27 04:57:36,019 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:36,019 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:36,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:36,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:36,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:36,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:36,020 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:36,020 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:36,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:36,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:36,021 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:36,021 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:36,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:36,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:36,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:36,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:36,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b74702a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:36,022 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:36,022 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:36,022 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:36,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:36,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:36,023 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:36,023 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:36,023 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:36,023 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,023 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:36,023 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,023 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:36,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:36,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:36,023 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:36,024 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:36,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:36,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:36,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:36,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:36,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:36,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:36,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:36,026 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:36,026 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:36,026 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:36,026 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:36,026 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:36,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:36,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:36,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:36,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:36,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:36,027 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:36,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:36,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:36,028 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:36,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:36,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,030 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,030 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:36,030 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:36,030 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:36,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:36,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:36,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:36,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:36,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:36,030 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:36,030 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:36,031 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,031 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:36,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:36,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:36,031 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:36,031 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:36,031 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:36,031 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:36,131 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:36,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:36,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:36,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:36,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:36,134 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:36,134 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:36,134 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:36,134 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:36,134 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:36,134 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:36,135 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,135 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:36,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:36,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:36,135 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,135 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:36,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:36,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:36,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:36,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:36,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:36,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:36,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:36,136 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:36,136 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:36,136 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:36,136 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:36,136 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:36,136 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:36,136 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:36,136 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-27 04:57:36,137 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:36,137 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:36,137 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:36,137 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:36,137 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:36,137 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:36,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:36,276 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:36,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679893056296, max timestamp=1679893056313Num uniques 859; 2023-03-27 04:57:36,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-27 04:57:36,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679893056296, max timestamp=1679893056313Num uniques 859; 2023-03-27 04:57:36,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679893056296, max timestamp=1679893056313Num uniques 859; 2023-03-27 04:57:36,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-27 04:57:36,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-27 04:57:36,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:36,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893116342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:36,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893116342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:36,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893116343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:36,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893116450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:36,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893116451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:36,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893116451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,551 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303271dd8b8f6840b44329473c876e03056bb to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303271dd8b8f6840b44329473c876e03056bb 2023-03-27 04:57:36,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:36,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893116656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:36,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893116657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:36,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893116669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,955 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/77d099bab9ff42fbaff40ec9ec172aed 2023-03-27 04:57:36,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:36,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893116960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:36,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893116961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:36,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:36,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893116974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,364 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/fa1034e960de47f09430d02879f13085 2023-03-27 04:57:37,374 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/74d2864b351c434087c9b40e97ddd5c2 2023-03-27 04:57:37,378 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/77d099bab9ff42fbaff40ec9ec172aed as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/77d099bab9ff42fbaff40ec9ec172aed 2023-03-27 04:57:37,381 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/77d099bab9ff42fbaff40ec9ec172aed, entries=100, sequenceid=98, filesize=18.4 K 2023-03-27 04:57:37,382 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/fa1034e960de47f09430d02879f13085 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/fa1034e960de47f09430d02879f13085 2023-03-27 04:57:37,384 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/fa1034e960de47f09430d02879f13085, entries=100, sequenceid=98, filesize=9.3 K 2023-03-27 04:57:37,385 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/74d2864b351c434087c9b40e97ddd5c2 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/74d2864b351c434087c9b40e97ddd5c2 2023-03-27 04:57:37,388 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/74d2864b351c434087c9b40e97ddd5c2, entries=100, sequenceid=98, filesize=9.3 K 2023-03-27 04:57:37,389 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 682db1751885437677f00c4ab8bf8ffc in 1253ms, sequenceid=98, compaction requested=true 2023-03-27 04:57:37,389 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:37,389 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:37,389 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:37,389 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:37,389 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,389 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:37,389 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:37,390 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,390 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,390 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:37,391 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:37,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:37,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:37,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:37,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:37,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:37,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c84009f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:37,392 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:37,392 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,393 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:37,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:37,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:37,393 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,393 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:37,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:37,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:37,393 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:37,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:37,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:37,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:37,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:37,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:37,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:37,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:37,397 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:37,397 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:37,397 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:37,397 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:37,397 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:37,398 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,398 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:37,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,398 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:37,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:37,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:37,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:37,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:37,398 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:37,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,399 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:37,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,401 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,401 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,401 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,401 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:37,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:37,401 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,401 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:37,402 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,402 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,402 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:37,402 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:37,402 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:37,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:37,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:37,402 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:37,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:37,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:37,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:37,484 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=154.31 KB heapSize=436.08 KB 2023-03-27 04:57:37,484 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:37,484 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-27 04:57:37,484 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:37,484 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-27 04:57:37,484 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:37,484 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-27 04:57:37,496 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327a82405624931403d895047dee00246dd to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327a82405624931403d895047dee00246dd 2023-03-27 04:57:37,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:37,501 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=136, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/11469f5721004b489903e1bc8616fee5 2023-03-27 04:57:37,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893117493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:37,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893117494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:37,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893117495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,502 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:37,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:37,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:37,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:37,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:37,505 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:37,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:37,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:37,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:37,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,505 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:37,506 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:37,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:37,506 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,506 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:37,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:37,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:37,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:37,507 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:37,507 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:37,507 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:37,507 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:37,507 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:37,507 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:37,507 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:37,507 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:37,507 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:37,507 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:37,507 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:37,507 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:37,507 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,507 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:37,508 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:37,508 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,508 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:37,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:37,509 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,509 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:37,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:37,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:37,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:37,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:37,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cb7ff4e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:37,510 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:37,510 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,510 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,510 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:37,511 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,511 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:37,511 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:37,511 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:37,511 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:37,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:37,512 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:37,512 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:37,512 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:37,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:37,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:37,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:37,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:37,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:37,514 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:37,514 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:37,514 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:37,514 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:37,514 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:37,515 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,515 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,515 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:37,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:37,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:37,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:37,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:37,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:37,516 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:37,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,517 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:37,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:37,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:37,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:37,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,518 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,519 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,519 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,519 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,519 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:37,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:37,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:37,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:37,520 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:37,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:37,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:37,520 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:37,520 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:37,520 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/0dff9f56ec1e48328093cbe4e53508ca 2023-03-27 04:57:37,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:37,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893117602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:37,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893117602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:37,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893117602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,620 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:37,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:37,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:37,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:37,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:37,623 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:37,623 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:37,623 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:37,623 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:37,623 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,623 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,624 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:37,624 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:37,624 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,624 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:37,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:37,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:37,625 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:37,625 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:37,625 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:37,625 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:37,625 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:37,625 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:37,625 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:37,625 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:37,625 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:37,625 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:37,625 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:37,625 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,625 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:37,625 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:37,626 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,626 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:37,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:37,626 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,626 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:37,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:37,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:37,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:37,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:37,627 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:37,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c4ae365[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:37,627 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,628 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:37,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:37,628 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:37,628 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,628 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:37,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,629 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:37,629 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:37,629 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:37,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:37,631 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:37,631 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:37,631 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:37,631 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:37,631 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:37,632 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,632 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:37,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,632 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:37,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:37,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:37,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:37,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:37,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,633 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,633 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:37,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,635 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:37,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:37,635 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,635 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:37,635 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:37,635 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:37,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:37,635 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:37,635 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:37,635 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,635 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:37,635 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,635 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,635 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:37,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:37,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:37,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893117668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,672 DEBUG [Thread-2298] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc., hostname=jenkins-hbase20.apache.org,44923,1679892955505, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:57:37,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:37,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893117677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,682 DEBUG [Thread-2294] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc., hostname=jenkins-hbase20.apache.org,44923,1679892955505, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:57:37,735 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:37,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:37,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:37,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:37,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:37,738 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:37,738 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:37,738 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:37,738 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:37,738 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,738 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,739 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,739 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:37,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:37,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:37,739 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,739 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:37,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:37,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:37,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:37,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:37,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:37,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:37,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:37,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:37,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:37,740 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:37,740 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:37,740 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:37,740 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:37,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:37,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:37,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:37,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:37,741 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:37,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:37,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:37,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:37,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:37,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:37,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:37,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54d731e6[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:37,743 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:37,743 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,743 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,743 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:37,743 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:37,743 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:37,743 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,743 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:37,744 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,744 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:37,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:37,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:37,744 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:37,744 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:37,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:37,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:37,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:37,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:37,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:37,746 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:37,746 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:37,746 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:37,746 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:37,746 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:37,747 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,747 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:37,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:37,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,747 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:37,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:37,747 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:37,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:37,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:37,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,748 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:37,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,749 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:37,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:37,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:37,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:37,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:37,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:37,750 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:37,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,750 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:37,750 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:37,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:37,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:37,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:37,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34614 deadline: 1679893117803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:37,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893117804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:37,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893117804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,850 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:37,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:37,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:37,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:37,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:37,854 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:37,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:37,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:37,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:37,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,855 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,855 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:37,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:37,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:37,856 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,856 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:37,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:37,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:37,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:37,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:37,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:37,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:37,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:37,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:37,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:37,857 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:37,857 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:37,857 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:37,857 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:37,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:37,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:37,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:37,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:37,858 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:37,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:37,858 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:37,858 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,858 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:37,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:37,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:37,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:37,859 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:37,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58db431b[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:37,859 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,860 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,860 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:37,860 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,860 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:37,860 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:37,860 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:37,860 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:37,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:37,860 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:37,861 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:37,861 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:37,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:37,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:37,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:37,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:37,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:37,862 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:37,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:37,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:37,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:37,862 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:37,863 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,863 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:37,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:37,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:37,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,864 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:37,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:37,865 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:37,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:37,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,865 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:37,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:37,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,867 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,867 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,867 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,867 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:37,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:37,867 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:37,867 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,867 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:37,867 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:37,867 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:37,867 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:37,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:37,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:37,867 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:37,867 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:37,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:37,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:37,929 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/5bc8def1bca64d89a1afa121b653efb0 2023-03-27 04:57:37,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/11469f5721004b489903e1bc8616fee5 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/11469f5721004b489903e1bc8616fee5 2023-03-27 04:57:37,935 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/11469f5721004b489903e1bc8616fee5, entries=250, sequenceid=136, filesize=38.9 K 2023-03-27 04:57:37,935 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/0dff9f56ec1e48328093cbe4e53508ca as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/0dff9f56ec1e48328093cbe4e53508ca 2023-03-27 04:57:37,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/0dff9f56ec1e48328093cbe4e53508ca, entries=150, sequenceid=136, filesize=11.8 K 2023-03-27 04:57:37,939 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/5bc8def1bca64d89a1afa121b653efb0 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/5bc8def1bca64d89a1afa121b653efb0 2023-03-27 04:57:37,942 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/5bc8def1bca64d89a1afa121b653efb0, entries=150, sequenceid=136, filesize=11.8 K 2023-03-27 04:57:37,942 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~436.03 KB/446496, currentSize=26.84 KB/27480 for 682db1751885437677f00c4ab8bf8ffc in 458ms, sequenceid=136, compaction requested=true 2023-03-27 04:57:37,942 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:37,942 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:37,942 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:57:37,942 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:57:37,942 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:37,943 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:37,943 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110223 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:57:37,944 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45453 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:57:37,944 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 682db1751885437677f00c4ab8bf8ffc/A is initiating minor compaction (all files) 2023-03-27 04:57:37,944 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 682db1751885437677f00c4ab8bf8ffc/B is initiating minor compaction (all files) 2023-03-27 04:57:37,944 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 682db1751885437677f00c4ab8bf8ffc/A in TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:37,944 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 682db1751885437677f00c4ab8bf8ffc/B in TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:37,944 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/218903ba93c84d238678c09ada4bdf47, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/f8b64f2bbe704392b8ba5182f11c03ff, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/77d099bab9ff42fbaff40ec9ec172aed, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/11469f5721004b489903e1bc8616fee5] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp, totalSize=107.6 K 2023-03-27 04:57:37,944 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/3022553b40534628aa5ff28a957ce511, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/faea3cab2a64481592b10dca3fe8de41, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/fa1034e960de47f09430d02879f13085, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/0dff9f56ec1e48328093cbe4e53508ca] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp, totalSize=44.4 K 2023-03-27 04:57:37,944 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 218903ba93c84d238678c09ada4bdf47, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679893053424 2023-03-27 04:57:37,944 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 3022553b40534628aa5ff28a957ce511, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679893053424 2023-03-27 04:57:37,944 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting f8b64f2bbe704392b8ba5182f11c03ff, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679893053500 2023-03-27 04:57:37,945 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting faea3cab2a64481592b10dca3fe8de41, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679893053500 2023-03-27 04:57:37,945 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 77d099bab9ff42fbaff40ec9ec172aed, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679893055647 2023-03-27 04:57:37,945 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting fa1034e960de47f09430d02879f13085, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679893055647 2023-03-27 04:57:37,945 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 11469f5721004b489903e1bc8616fee5, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679893056321 2023-03-27 04:57:37,945 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 0dff9f56ec1e48328093cbe4e53508ca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679893056341 2023-03-27 04:57:37,958 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 682db1751885437677f00c4ab8bf8ffc#B#compaction#320 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:37,961 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 682db1751885437677f00c4ab8bf8ffc#A#compaction#319 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:37,967 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:37,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:37,970 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/2fcdaf36d5a14acc88836fd3e7f677da as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/2fcdaf36d5a14acc88836fd3e7f677da 2023-03-27 04:57:37,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:37,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:37,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:37,973 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:37,973 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:37,973 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:37,973 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:37,974 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,974 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,975 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:37,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:37,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:37,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,976 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:37,976 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:37,976 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:37,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:37,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:37,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:37,978 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 682db1751885437677f00c4ab8bf8ffc/B of 682db1751885437677f00c4ab8bf8ffc into 2fcdaf36d5a14acc88836fd3e7f677da(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:37,978 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:37,978 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc., storeName=682db1751885437677f00c4ab8bf8ffc/B, priority=12, startTime=1679893057942; duration=0sec 2023-03-27 04:57:37,978 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:37,978 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:57:37,979 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45453 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:57:37,979 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 682db1751885437677f00c4ab8bf8ffc/C is initiating minor compaction (all files) 2023-03-27 04:57:37,979 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 682db1751885437677f00c4ab8bf8ffc/C in TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:37,979 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8c575df33b18447ea1aba69b8c7e287c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/2ef90a4bbdc747aaa51a7351d23ab60d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/74d2864b351c434087c9b40e97ddd5c2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/5bc8def1bca64d89a1afa121b653efb0] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp, totalSize=44.4 K 2023-03-27 04:57:37,980 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 8c575df33b18447ea1aba69b8c7e287c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679893053424 2023-03-27 04:57:37,980 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 2ef90a4bbdc747aaa51a7351d23ab60d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679893053500 2023-03-27 04:57:37,980 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 74d2864b351c434087c9b40e97ddd5c2, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679893055647 2023-03-27 04:57:37,980 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 5bc8def1bca64d89a1afa121b653efb0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679893056341 2023-03-27 04:57:37,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:37,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:37,982 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:37,982 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:37,982 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:37,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:37,986 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:37,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:37,986 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:37,986 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-27 04:57:37,987 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:37,987 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:37,987 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:37,987 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:37,987 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:37,987 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:37,988 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 682db1751885437677f00c4ab8bf8ffc#C#compaction#321 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:38,036 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/d400e690a98d4776b7805223b5bb40b7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/d400e690a98d4776b7805223b5bb40b7 2023-03-27 04:57:38,040 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 682db1751885437677f00c4ab8bf8ffc/C of 682db1751885437677f00c4ab8bf8ffc into d400e690a98d4776b7805223b5bb40b7(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:38,040 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:38,040 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc., storeName=682db1751885437677f00c4ab8bf8ffc/C, priority=12, startTime=1679893057943; duration=0sec 2023-03-27 04:57:38,040 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:38,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:38,115 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:38,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893058132, max timestamp=1679893058144Num uniques 859; 2023-03-27 04:57:38,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:57:38,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893058132, max timestamp=1679893058144Num uniques 859; 2023-03-27 04:57:38,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893058132, max timestamp=1679893058144Num uniques 859; 2023-03-27 04:57:38,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:57:38,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:57:38,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679893058160, max timestamp=1679893058184Num uniques 1068; 2023-03-27 04:57:38,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-27 04:57:38,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679893058160, max timestamp=1679893058184Num uniques 1068; 2023-03-27 04:57:38,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679893058160, max timestamp=1679893058184Num uniques 1068; 2023-03-27 04:57:38,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-27 04:57:38,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-27 04:57:38,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679893058204, max timestamp=1679893058214Num uniques 1077; 2023-03-27 04:57:38,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=14972, suffix heap size=139976,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=327 2023-03-27 04:57:38,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679893058204, max timestamp=1679893058214Num uniques 1077; 2023-03-27 04:57:38,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679893058204, max timestamp=1679893058214Num uniques 1077; 2023-03-27 04:57:38,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=14972, suffix heap size=139976,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=327 2023-03-27 04:57:38,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=14972, suffix heap size=139976,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=327 2023-03-27 04:57:38,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:38,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34588 deadline: 1679893118248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:38,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679893058233, max timestamp=1679893058253Num uniques 1186; 2023-03-27 04:57:38,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=10806, suffix heap size=151312,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=236 2023-03-27 04:57:38,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679893058233, max timestamp=1679893058253Num uniques 1186; 2023-03-27 04:57:38,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=10806, suffix heap size=151312,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=236 2023-03-27 04:57:38,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679893058233, max timestamp=1679893058253Num uniques 1186; 2023-03-27 04:57:38,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=10806, suffix heap size=151312,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=236 2023-03-27 04:57:38,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679893058288, max timestamp=1679893058315Num uniques 1095; 2023-03-27 04:57:38,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679893058288, max timestamp=1679893058315Num uniques 1095; 2023-03-27 04:57:38,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-27 04:57:38,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-27 04:57:38,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679893058288, max timestamp=1679893058315Num uniques 1095; 2023-03-27 04:57:38,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-27 04:57:38,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:38,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34586 deadline: 1679893118356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:38,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679893058344, max timestamp=1679893058356Num uniques 1104; 2023-03-27 04:57:38,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679893058344, max timestamp=1679893058356Num uniques 1104; 2023-03-27 04:57:38,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-27 04:57:38,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679893058344, max timestamp=1679893058356Num uniques 1104; 2023-03-27 04:57:38,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-27 04:57:38,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-27 04:57:38,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679893058375, max timestamp=1679893058412Num uniques 1113; 2023-03-27 04:57:38,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679893058375, max timestamp=1679893058412Num uniques 1063; 2023-03-27 04:57:38,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-27 04:57:38,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-03-27 04:57:38,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679893058375, max timestamp=1679893058412Num uniques 1063; 2023-03-27 04:57:38,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-03-27 04:57:38,445 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327e5eaacb2a470487a91354c15ec0f5d9e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e5eaacb2a470487a91354c15ec0f5d9e 2023-03-27 04:57:38,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679893058451, max timestamp=1679893058471Num uniques 1072; 2023-03-27 04:57:38,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-27 04:57:38,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679893058451, max timestamp=1679893058471Num uniques 1072; 2023-03-27 04:57:38,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679893058451, max timestamp=1679893058471Num uniques 1072; 2023-03-27 04:57:38,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-27 04:57:38,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-27 04:57:38,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679893058489, max timestamp=1679893058499Num uniques 1081; 2023-03-27 04:57:38,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-03-27 04:57:38,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679893058489, max timestamp=1679893058499Num uniques 1081; 2023-03-27 04:57:38,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679893058489, max timestamp=1679893058499Num uniques 1081; 2023-03-27 04:57:38,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-03-27 04:57:38,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-03-27 04:57:38,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679893058522, max timestamp=1679893058538Num uniques 1140; 2023-03-27 04:57:38,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=13280, suffix heap size=146528,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=290 2023-03-27 04:57:38,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679893058522, max timestamp=1679893058538Num uniques 1140; 2023-03-27 04:57:38,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=13280, suffix heap size=146528,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=290 2023-03-27 04:57:38,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679893058522, max timestamp=1679893058538Num uniques 1140; 2023-03-27 04:57:38,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=13280, suffix heap size=146528,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=290 2023-03-27 04:57:38,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=399, cellSize=18274, totalHeapSize=41736, min timestamp=1679893058558, max timestamp=1679893058578Num uniques 1149; 2023-03-27 04:57:38,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=18274, suffix heap size=147464,new segment heap size=41736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=399 2023-03-27 04:57:38,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=399, cellSize=18274, totalHeapSize=41736, min timestamp=1679893058558, max timestamp=1679893058578Num uniques 1149; 2023-03-27 04:57:38,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=18274, suffix heap size=147464,new segment heap size=41736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=399 2023-03-27 04:57:38,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=399, cellSize=18274, totalHeapSize=41736, min timestamp=1679893058558, max timestamp=1679893058578Num uniques 1149; 2023-03-27 04:57:38,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=18274, suffix heap size=147464,new segment heap size=41736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=399 2023-03-27 04:57:38,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:38,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893118612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:38,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:38,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893118618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:38,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:38,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893118619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:38,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:38,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893118720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:38,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:38,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893118730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:38,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:38,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893118731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:38,787 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/6414ceb30694420bb1931ea243db30e7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/6414ceb30694420bb1931ea243db30e7 2023-03-27 04:57:38,789 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 682db1751885437677f00c4ab8bf8ffc/A of 682db1751885437677f00c4ab8bf8ffc into 6414ceb30694420bb1931ea243db30e7(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:38,790 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:38,790 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc., storeName=682db1751885437677f00c4ab8bf8ffc/A, priority=12, startTime=1679893057942; duration=0sec 2023-03-27 04:57:38,790 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:38,850 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=144, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/999d816008c34fe7a21d4506fdba4f95 2023-03-27 04:57:38,858 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/15803a67f0324cfca3aa862d8a244787 2023-03-27 04:57:38,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:38,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893118924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:38,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:38,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893118935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:38,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:38,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893118935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:39,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893119229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:39,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893119241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:39,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893119242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,268 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/070690cfb599475cb5dc5f67a374f223 2023-03-27 04:57:39,271 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/999d816008c34fe7a21d4506fdba4f95 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/999d816008c34fe7a21d4506fdba4f95 2023-03-27 04:57:39,273 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/999d816008c34fe7a21d4506fdba4f95, entries=150, sequenceid=144, filesize=25.3 K 2023-03-27 04:57:39,274 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/15803a67f0324cfca3aa862d8a244787 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/15803a67f0324cfca3aa862d8a244787 2023-03-27 04:57:39,277 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/15803a67f0324cfca3aa862d8a244787, entries=150, sequenceid=144, filesize=11.8 K 2023-03-27 04:57:39,277 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/070690cfb599475cb5dc5f67a374f223 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/070690cfb599475cb5dc5f67a374f223 2023-03-27 04:57:39,281 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/070690cfb599475cb5dc5f67a374f223, entries=150, sequenceid=144, filesize=11.8 K 2023-03-27 04:57:39,282 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 682db1751885437677f00c4ab8bf8ffc in 1296ms, sequenceid=144, compaction requested=false 2023-03-27 04:57:39,282 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:39,282 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:39,282 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:39,282 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:39,282 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,282 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:39,282 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:39,283 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:39,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:39,283 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:39,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:39,283 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:39,283 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:39,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:39,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:39,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:39,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:39,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2be59ac1[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:39,284 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:39,284 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:39,285 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:39,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:39,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:39,285 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:39,285 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:39,285 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:39,285 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,285 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:39,286 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:39,286 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,286 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:39,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:39,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:39,286 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:39,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:39,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:39,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:39,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:39,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:39,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:39,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:39,288 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:39,288 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:39,288 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:39,288 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:39,288 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:39,289 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:39,289 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:39,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:39,289 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:39,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:39,289 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:39,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:39,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,290 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:39,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:39,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,291 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,291 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:39,291 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:39,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:39,292 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:39,291 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:39,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:39,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:39,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:39,292 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:39,292 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,292 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:39,292 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:39,292 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:39,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:39,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:39,292 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:39,292 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:39,392 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:39,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:39,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:39,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:39,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:39,395 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:39,395 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:39,395 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:39,395 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:39,395 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:39,395 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:39,396 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,396 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:39,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:39,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:39,396 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,396 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:39,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:39,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:39,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:39,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:39,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:39,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:39,397 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:39,397 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:39,397 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:39,397 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:39,397 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:39,397 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:39,397 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:39,397 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=161.02 KB heapSize=442.17 KB 2023-03-27 04:57:39,398 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:39,398 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:39,398 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:39,398 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:39,398 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:39,398 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:39,410 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327be0ca5c201c94fb695f37451aaaaf7f6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327be0ca5c201c94fb695f37451aaaaf7f6 2023-03-27 04:57:39,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:39,740 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:39,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:39,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893119748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:39,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893119749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:39,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893119750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,814 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=354, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/256c32ec99ae46b7be596d3e27581608 2023-03-27 04:57:39,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:39,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893119854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:39,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:39,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893119856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:40,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893120060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:40,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893120060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,224 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/0effd4bc255a403ca0be8f27e614e032 2023-03-27 04:57:40,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:40,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893120362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:40,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893120364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,633 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/86c6ba31da93457c865cec7399c22a28 2023-03-27 04:57:40,637 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/256c32ec99ae46b7be596d3e27581608 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/256c32ec99ae46b7be596d3e27581608 2023-03-27 04:57:40,640 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/256c32ec99ae46b7be596d3e27581608, entries=150, sequenceid=354, filesize=25.4 K 2023-03-27 04:57:40,641 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/0effd4bc255a403ca0be8f27e614e032 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/0effd4bc255a403ca0be8f27e614e032 2023-03-27 04:57:40,644 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/0effd4bc255a403ca0be8f27e614e032, entries=150, sequenceid=354, filesize=11.9 K 2023-03-27 04:57:40,645 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/86c6ba31da93457c865cec7399c22a28 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/86c6ba31da93457c865cec7399c22a28 2023-03-27 04:57:40,647 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/86c6ba31da93457c865cec7399c22a28, entries=150, sequenceid=354, filesize=11.9 K 2023-03-27 04:57:40,650 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~442.13 KB/452736, currentSize=26.84 KB/27480 for 682db1751885437677f00c4ab8bf8ffc in 1253ms, sequenceid=354, compaction requested=true 2023-03-27 04:57:40,650 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:40,650 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:40,650 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:40,650 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:40,650 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,651 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:40,651 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:40,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:40,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:40,652 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:40,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:40,652 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:40,652 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:40,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:40,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:40,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:40,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:40,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40df1513[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:40,653 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:40,653 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:40,653 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:40,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:40,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:40,653 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:40,653 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:40,653 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,654 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:40,654 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:40,654 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,654 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:40,654 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:40,654 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:40,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:40,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:40,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:40,657 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:40,657 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:40,657 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:40,657 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:40,657 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:40,658 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:40,658 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:40,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:40,658 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:40,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:40,658 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,659 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:40,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:40,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,661 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,661 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:40,661 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:40,661 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:40,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:40,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:40,661 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:40,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:40,661 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:40,661 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:40,661 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:40,661 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,662 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:40,662 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:40,662 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:40,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:40,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:40,762 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:40,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:40,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:40,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:40,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:40,765 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:40,765 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:40,765 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:40,765 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:40,765 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:40,765 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:40,766 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,766 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:40,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:40,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:40,766 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,766 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:40,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:40,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:40,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:40,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:40,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:40,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:40,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:40,767 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:40,767 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:40,767 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:40,767 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:40,767 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:40,767 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:40,767 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:40,767 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-27 04:57:40,767 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:40,768 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:40,768 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:40,768 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:40,768 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:40,768 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:40,779 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032732d007d04a7d4d7396314a06d123c22d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032732d007d04a7d4d7396314a06d123c22d 2023-03-27 04:57:40,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:40,810 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:40,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893060855, max timestamp=1679893060868Num uniques 859; 2023-03-27 04:57:40,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893060855, max timestamp=1679893060868Num uniques 859; 2023-03-27 04:57:40,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893060855, max timestamp=1679893060868Num uniques 859; 2023-03-27 04:57:40,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:57:40,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:57:40,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:57:40,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679893060884, max timestamp=1679893060916Num uniques 1018; 2023-03-27 04:57:40,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-27 04:57:40,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679893060884, max timestamp=1679893060916Num uniques 1018; 2023-03-27 04:57:40,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-27 04:57:40,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679893060884, max timestamp=1679893060916Num uniques 1018; 2023-03-27 04:57:40,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-27 04:57:40,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:40,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893120953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:40,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679893060948, max timestamp=1679893060957Num uniques 1127; 2023-03-27 04:57:40,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-03-27 04:57:40,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679893060948, max timestamp=1679893060957Num uniques 1127; 2023-03-27 04:57:40,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679893060948, max timestamp=1679893060957Num uniques 1127; 2023-03-27 04:57:40,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-03-27 04:57:40,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-27 04:57:41,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679893060987, max timestamp=1679893061017Num uniques 1086; 2023-03-27 04:57:41,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-27 04:57:41,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679893060987, max timestamp=1679893061017Num uniques 1086; 2023-03-27 04:57:41,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-27 04:57:41,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679893060987, max timestamp=1679893061017Num uniques 1036; 2023-03-27 04:57:41,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-27 04:57:41,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679893061051, max timestamp=1679893061058Num uniques 1045; 2023-03-27 04:57:41,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-27 04:57:41,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679893061051, max timestamp=1679893061058Num uniques 1045; 2023-03-27 04:57:41,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-27 04:57:41,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679893061051, max timestamp=1679893061058Num uniques 1045; 2023-03-27 04:57:41,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-27 04:57:41,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:41,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893121088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:41,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:41,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893121088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:41,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679893061080, max timestamp=1679893061092Num uniques 1104; 2023-03-27 04:57:41,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679893061080, max timestamp=1679893061092Num uniques 1104; 2023-03-27 04:57:41,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-27 04:57:41,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-27 04:57:41,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679893061080, max timestamp=1679893061092Num uniques 1104; 2023-03-27 04:57:41,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-27 04:57:41,183 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=361, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/dac8264768564a3786e6776ae8510137 2023-03-27 04:57:41,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679893061154, max timestamp=1679893061187Num uniques 1063; 2023-03-27 04:57:41,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-03-27 04:57:41,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679893061154, max timestamp=1679893061187Num uniques 1063; 2023-03-27 04:57:41,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679893061154, max timestamp=1679893061187Num uniques 1063; 2023-03-27 04:57:41,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-03-27 04:57:41,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-03-27 04:57:41,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679893061222, max timestamp=1679893061236Num uniques 1072; 2023-03-27 04:57:41,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-27 04:57:41,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679893061222, max timestamp=1679893061236Num uniques 1072; 2023-03-27 04:57:41,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-27 04:57:41,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679893061222, max timestamp=1679893061236Num uniques 1072; 2023-03-27 04:57:41,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-27 04:57:41,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679893061281, max timestamp=1679893061295Num uniques 1081; 2023-03-27 04:57:41,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-27 04:57:41,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679893061281, max timestamp=1679893061295Num uniques 1081; 2023-03-27 04:57:41,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679893061281, max timestamp=1679893061295Num uniques 1081; 2023-03-27 04:57:41,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-27 04:57:41,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-27 04:57:41,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679893061320, max timestamp=1679893061334Num uniques 1090; 2023-03-27 04:57:41,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-03-27 04:57:41,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679893061320, max timestamp=1679893061334Num uniques 1090; 2023-03-27 04:57:41,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-03-27 04:57:41,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679893061320, max timestamp=1679893061334Num uniques 1090; 2023-03-27 04:57:41,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-03-27 04:57:41,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:41,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893121371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:41,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679893061357, max timestamp=1679893061372Num uniques 1149; 2023-03-27 04:57:41,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-03-27 04:57:41,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679893061357, max timestamp=1679893061372Num uniques 1149; 2023-03-27 04:57:41,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-03-27 04:57:41,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679893061357, max timestamp=1679893061372Num uniques 1149; 2023-03-27 04:57:41,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-03-27 04:57:41,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679893061405, max timestamp=1679893061419Num uniques 1108; 2023-03-27 04:57:41,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-03-27 04:57:41,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679893061405, max timestamp=1679893061419Num uniques 1108; 2023-03-27 04:57:41,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-03-27 04:57:41,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679893061405, max timestamp=1679893061419Num uniques 1108; 2023-03-27 04:57:41,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-03-27 04:57:41,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679893061468, max timestamp=1679893061481Num uniques 1067; 2023-03-27 04:57:41,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679893061468, max timestamp=1679893061481Num uniques 1067; 2023-03-27 04:57:41,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679893061468, max timestamp=1679893061481Num uniques 1067; 2023-03-27 04:57:41,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-27 04:57:41,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-27 04:57:41,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-27 04:57:41,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1679893061509, max timestamp=1679893061518Num uniques 1026; 2023-03-27 04:57:41,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=12636, suffix heap size=134672,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=276 2023-03-27 04:57:41,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1679893061509, max timestamp=1679893061518Num uniques 1026; 2023-03-27 04:57:41,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1679893061509, max timestamp=1679893061518Num uniques 1026; 2023-03-27 04:57:41,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=12636, suffix heap size=134672,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=276 2023-03-27 04:57:41,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=12636, suffix heap size=134672,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=276 2023-03-27 04:57:41,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:41,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893121547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:41,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:41,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893121553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:41,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679893061540, max timestamp=1679893061553Num uniques 1135; 2023-03-27 04:57:41,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679893061540, max timestamp=1679893061553Num uniques 1135; 2023-03-27 04:57:41,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=10760, suffix heap size=146008,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=235 2023-03-27 04:57:41,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679893061540, max timestamp=1679893061553Num uniques 1135; 2023-03-27 04:57:41,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=10760, suffix heap size=146008,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=235 2023-03-27 04:57:41,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=10760, suffix heap size=146008,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=235 2023-03-27 04:57:41,592 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/229f1a1679a34637ae3b21135e989419 2023-03-27 04:57:41,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679893061612, max timestamp=1679893061626Num uniques 1094; 2023-03-27 04:57:41,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679893061612, max timestamp=1679893061626Num uniques 1094; 2023-03-27 04:57:41,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679893061612, max timestamp=1679893061626Num uniques 1094; 2023-03-27 04:57:41,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-03-27 04:57:41,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-03-27 04:57:41,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-03-27 04:57:41,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679893061669, max timestamp=1679893061688Num uniques 1053; 2023-03-27 04:57:41,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679893061669, max timestamp=1679893061688Num uniques 1053; 2023-03-27 04:57:41,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679893061669, max timestamp=1679893061688Num uniques 1053; 2023-03-27 04:57:41,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-27 04:57:41,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-27 04:57:41,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-27 04:57:41,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1679893061706, max timestamp=1679893061715Num uniques 1012; 2023-03-27 04:57:41,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-03-27 04:57:41,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1679893061706, max timestamp=1679893061715Num uniques 1012; 2023-03-27 04:57:41,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-03-27 04:57:41,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1679893061706, max timestamp=1679893061715Num uniques 1012; 2023-03-27 04:57:41,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-03-27 04:57:41,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:41,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893121750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:41,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1679893061737, max timestamp=1679893061755Num uniques 1121; 2023-03-27 04:57:41,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=7826, suffix heap size=144552,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=171 2023-03-27 04:57:41,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1679893061737, max timestamp=1679893061755Num uniques 1121; 2023-03-27 04:57:41,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1679893061737, max timestamp=1679893061755Num uniques 1121; 2023-03-27 04:57:41,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=7826, suffix heap size=144552,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=171 2023-03-27 04:57:41,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=7826, suffix heap size=144552,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=171 2023-03-27 04:57:41,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1679893061774, max timestamp=1679893061786Num uniques 1030; 2023-03-27 04:57:41,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1679893061774, max timestamp=1679893061786Num uniques 1030; 2023-03-27 04:57:41,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-03-27 04:57:41,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1679893061774, max timestamp=1679893061786Num uniques 1030; 2023-03-27 04:57:41,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-03-27 04:57:41,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-03-27 04:57:41,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1679893061813, max timestamp=1679893061844Num uniques 1039; 2023-03-27 04:57:41,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=13234, suffix heap size=136024,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=289 2023-03-27 04:57:41,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1679893061813, max timestamp=1679893061844Num uniques 1039; 2023-03-27 04:57:41,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1679893061813, max timestamp=1679893061844Num uniques 1039; 2023-03-27 04:57:41,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=13234, suffix heap size=136024,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=289 2023-03-27 04:57:41,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=8654, suffix heap size=136024,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=189 2023-03-27 04:57:41,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=398, cellSize=18228, totalHeapSize=41632, min timestamp=1679893061877, max timestamp=1679893061888Num uniques 1148; 2023-03-27 04:57:41,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=18228, suffix heap size=147360,new segment heap size=41632 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=398 2023-03-27 04:57:41,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=398, cellSize=18228, totalHeapSize=41632, min timestamp=1679893061877, max timestamp=1679893061888Num uniques 1148; 2023-03-27 04:57:41,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=18228, suffix heap size=147360,new segment heap size=41632 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=398 2023-03-27 04:57:41,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=398, cellSize=18228, totalHeapSize=41632, min timestamp=1679893061877, max timestamp=1679893061888Num uniques 1048; 2023-03-27 04:57:41,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=18228, suffix heap size=136960,new segment heap size=41632 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=398 2023-03-27 04:57:41,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:41,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893121909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:41,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:41,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893121914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:41,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:41,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893121915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:41,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:41,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893121915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:41,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:41,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893121915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,000 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/c3dca3502476411c99de162cbf80b433 2023-03-27 04:57:42,003 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/dac8264768564a3786e6776ae8510137 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/dac8264768564a3786e6776ae8510137 2023-03-27 04:57:42,006 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/dac8264768564a3786e6776ae8510137, entries=100, sequenceid=361, filesize=18.6 K 2023-03-27 04:57:42,007 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/229f1a1679a34637ae3b21135e989419 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/229f1a1679a34637ae3b21135e989419 2023-03-27 04:57:42,009 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/229f1a1679a34637ae3b21135e989419, entries=100, sequenceid=361, filesize=9.5 K 2023-03-27 04:57:42,010 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/c3dca3502476411c99de162cbf80b433 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/c3dca3502476411c99de162cbf80b433 2023-03-27 04:57:42,012 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/c3dca3502476411c99de162cbf80b433, entries=100, sequenceid=361, filesize=9.5 K 2023-03-27 04:57:42,013 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 682db1751885437677f00c4ab8bf8ffc in 1246ms, sequenceid=361, compaction requested=true 2023-03-27 04:57:42,013 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:42,013 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:42,013 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:42,013 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:42,013 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,013 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:42,013 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:42,015 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,015 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,015 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,015 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:42,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d6a54ef[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:42,016 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:42,016 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,017 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,017 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:42,017 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:42,017 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:42,017 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:42,018 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,018 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:42,018 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,018 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:42,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:42,020 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:42,020 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:42,020 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:42,020 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:42,020 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:42,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:42,021 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=167.72 KB heapSize=462.19 KB 2023-03-27 04:57:42,021 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,021 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,021 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,022 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:42,022 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-27 04:57:42,022 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:42,022 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-27 04:57:42,022 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:42,022 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-27 04:57:42,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,024 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,025 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:42,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:42,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,027 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:42,027 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,027 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:42,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:42,027 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:42,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:42,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:42,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893122021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893122022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893122024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893122030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893122031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,127 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:42,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:42,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:42,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:42,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:42,130 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:42,130 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:42,130 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:42,130 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:42,130 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,131 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,131 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,131 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:42,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:42,131 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,131 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:42,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:42,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:42,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:42,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:42,132 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:42,132 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:42,132 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:42,133 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:42,133 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:42,133 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:42,133 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:42,133 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:42,133 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:42,133 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:42,133 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:42,133 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,133 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:42,133 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:42,134 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,134 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,134 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,134 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:42,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893122132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893122133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:42,136 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:42,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fab09c2[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:42,136 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,137 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,137 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:42,137 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:42,137 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:42,137 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,137 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:42,137 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,137 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,137 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,138 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:42,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:42,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:42,139 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:42,139 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:42,139 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:42,139 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:42,139 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:42,140 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,140 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,140 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,140 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893122138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,143 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,144 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,144 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:42,144 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,144 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:42,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:42,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:42,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:42,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:42,144 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,144 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,144 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,144 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:42,144 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,144 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,144 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:42,145 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893122232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893122233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,245 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:42,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:42,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:42,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:42,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:42,247 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:42,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:42,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:42,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:42,248 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,248 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,248 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:42,248 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:42,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:42,249 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,249 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:42,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:42,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:42,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:42,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:42,249 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:42,249 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:42,250 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:42,250 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:42,250 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:42,250 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:42,250 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:42,250 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:42,250 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:42,250 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:42,250 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:42,250 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,250 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:42,250 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:42,251 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,251 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,251 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,251 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:42,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@281fad64[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:42,252 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:42,252 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,253 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:42,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:42,253 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,253 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:42,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:42,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,254 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,254 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,254 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:42,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:42,256 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:42,256 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:42,256 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:42,256 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:42,256 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:42,256 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,257 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,257 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,258 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,260 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,261 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:42,261 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,262 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:42,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:42,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:42,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:42,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:42,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,262 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:42,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,262 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:42,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893122337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893122337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893122343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,362 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:42,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:42,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:42,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:42,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:42,365 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:42,365 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:42,365 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:42,365 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:42,365 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,365 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,366 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:42,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:42,366 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,367 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,367 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:42,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:42,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:42,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:42,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:42,368 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:42,368 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:42,368 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:42,368 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:42,368 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:42,368 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:42,368 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:42,368 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:42,368 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:42,368 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:42,368 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:42,368 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,368 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:42,368 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:42,369 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,369 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,369 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,369 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:42,370 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:42,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@105042f9[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:42,370 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,371 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,371 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:42,371 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,371 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:42,371 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:42,371 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:42,372 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,372 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:42,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,372 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,373 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:42,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:42,374 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:42,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:42,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:42,374 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:42,374 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:42,376 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,376 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,376 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,377 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,378 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:42,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:42,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,379 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,380 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,380 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:42,380 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:42,380 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:42,380 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:42,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,433 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327ae3868807f6c453eab15083d6c32d9f7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327ae3868807f6c453eab15083d6c32d9f7 2023-03-27 04:57:42,480 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:42,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:42,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:42,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:42,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:42,482 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:42,482 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:42,483 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:42,483 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:42,483 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,483 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,483 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,483 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:42,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:42,484 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,484 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:42,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:42,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:42,484 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:42,485 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:42,485 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:42,485 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:42,485 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:42,485 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:42,485 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:42,485 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:42,485 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:42,485 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:42,485 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:42,485 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,485 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:42,485 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:42,486 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,486 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,486 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,486 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:42,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fe2b130[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:42,487 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:42,487 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,488 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,488 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:42,488 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,488 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:42,488 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:42,488 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:42,488 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,488 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:42,489 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,489 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:42,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:42,490 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:42,490 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:42,490 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:42,490 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:42,490 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:42,491 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,491 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,491 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,492 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,492 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,494 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,494 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,494 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:42,494 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,494 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:42,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:42,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:42,494 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,494 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,494 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,494 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:42,494 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,495 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:42,495 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:42,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893122536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893122537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,595 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:42,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:42,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:42,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:42,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:42,598 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:42,598 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:42,598 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:42,598 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:42,598 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,598 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,599 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,599 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:42,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:42,599 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,599 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:42,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:42,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:42,600 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:42,600 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:42,600 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:42,600 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:42,600 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:42,600 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:42,600 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:42,600 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:42,600 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:42,600 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:42,600 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:42,600 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,600 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:42,600 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:42,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,601 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,602 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,602 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:42,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:42,603 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:42,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18705017[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:42,603 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,603 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,603 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:42,603 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,603 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:42,603 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:42,603 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:42,604 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,604 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:42,604 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,604 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:42,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:42,606 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:42,606 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:42,606 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:42,606 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:42,606 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:42,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,608 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,610 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,610 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,610 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:42,610 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:42,610 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,610 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:42,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:42,610 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,610 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,610 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,610 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:42,610 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:42,610 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:42,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:42,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893122640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893122641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:42,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893122647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,711 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:42,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:42,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:42,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:42,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:42,713 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:42,713 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:42,713 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:42,713 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:42,714 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,714 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,714 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,714 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:42,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:42,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:42,715 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,715 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:42,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:42,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:42,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:42,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:42,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:42,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:42,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:42,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:42,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:42,716 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:42,716 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:42,716 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:42,716 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:42,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:42,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:42,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:42,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:42,717 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,717 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,717 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,717 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:42,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:42,719 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:42,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d51baf0[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:42,719 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,719 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:42,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:42,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:42,719 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,719 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:42,720 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:42,720 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,720 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:42,720 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:42,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:42,723 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:42,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:42,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:42,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:42,723 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:42,725 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,725 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,725 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,725 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,726 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,727 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,728 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,728 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:42,728 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,728 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:42,728 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:42,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:42,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,728 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,728 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:42,728 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,728 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,728 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,728 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:42,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:42,828 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:42,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:42,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:42,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:42,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:42,831 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:42,831 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:42,831 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:42,831 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:42,832 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,832 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,832 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,832 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:42,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:42,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:42,833 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,833 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:42,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:42,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:42,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:42,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:42,834 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:42,834 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:42,834 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:42,834 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:42,834 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:42,834 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:42,834 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:42,834 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:42,834 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:42,834 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:42,834 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:42,834 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,834 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:42,834 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:42,835 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,835 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,835 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,835 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:42,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:42,837 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:42,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f046e29[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:42,837 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,837 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,837 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,837 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:42,837 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:42,837 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:42,837 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:42,838 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,838 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:42,838 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,838 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:42,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,840 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=760, memsize=58.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/2bd2bbb38ad14ebbb5680bfd1adae94e 2023-03-27 04:57:42,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:42,840 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:42,840 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:42,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:42,840 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:42,840 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:42,840 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:42,841 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,841 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,841 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,842 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,843 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,859 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,859 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:42,860 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:42,860 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:42,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:42,859 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:42,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,860 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,860 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:42,860 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,860 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:42,860 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,860 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,860 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,960 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:42,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:42,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:42,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:42,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:42,963 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:42,963 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:42,963 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:42,963 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:42,964 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,964 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,964 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,964 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:42,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:42,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:42,965 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,965 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:42,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:42,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:42,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:42,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:42,965 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:42,965 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:42,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:42,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:42,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:42,966 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:42,966 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:42,966 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:42,966 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:42,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:42,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:42,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:42,966 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:42,967 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,967 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,967 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,967 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:42,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:42,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25952fc8[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:42,968 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:42,968 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,969 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,969 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:42,969 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,969 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:42,969 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:42,969 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:42,970 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:42,970 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,970 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,970 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:42,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:42,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:42,971 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:42,971 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:42,971 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:42,972 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:42,972 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:42,972 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,972 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:42,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:42,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,972 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:42,973 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:42,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:42,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,974 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:42,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:42,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,975 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,975 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,975 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:42,975 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:42,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:42,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:42,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:42,975 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:42,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:42,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:42,975 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:42,975 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:42,976 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:42,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:42,976 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:42,976 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:42,976 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:42,976 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:43,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893123042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:43,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893123043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,076 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:43,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:43,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:43,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:43,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:43,078 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:43,078 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:43,078 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:43,078 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:43,079 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,079 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,079 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:43,079 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,080 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,080 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:43,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:43,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:43,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:43,080 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:43,081 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:43,081 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:43,081 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:43,081 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:43,081 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:43,081 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:43,081 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:43,081 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:43,081 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:43,081 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:43,081 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,081 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:43,081 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:43,082 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,082 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:43,082 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,082 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:43,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:43,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:43,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7f3385[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:43,083 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:43,083 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,084 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,084 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:43,084 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:43,084 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:43,084 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,084 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:43,084 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:43,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:43,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:43,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:43,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,085 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:43,085 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:43,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,085 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:43,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:43,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:43,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:43,088 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:43,088 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:43,088 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:43,088 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:43,088 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:43,093 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,093 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:43,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:43,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,093 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:43,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:43,093 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:43,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,094 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:43,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:43,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,096 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:43,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:43,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:43,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:43,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:43,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:43,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:43,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,097 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,097 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,097 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,097 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:43,097 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:43,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:43,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:43,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:43,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893123145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:43,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893123147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:43,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893123154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,197 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:43,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:43,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:43,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:43,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:43,199 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:43,199 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:43,200 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:43,200 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:43,200 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,200 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,200 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,200 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:43,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:43,201 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,201 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:43,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:43,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:43,204 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:43,204 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:43,204 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:43,204 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:43,204 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:43,204 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:43,204 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:43,204 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:43,204 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:43,204 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:43,205 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:43,205 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,205 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:43,205 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:43,206 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,206 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:43,206 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,206 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:43,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:43,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fb988a0[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:43,207 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:43,207 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,209 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,209 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:43,209 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:43,209 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:43,209 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,209 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:43,210 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,210 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:43,210 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:43,210 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:43,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:43,211 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:43,211 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:43,211 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:43,212 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:43,212 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:43,212 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,212 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,212 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:43,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,213 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:43,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,213 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:43,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,216 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,216 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,216 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,216 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:43,216 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,216 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,216 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:43,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:43,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:43,216 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:43,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:43,216 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,216 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:43,216 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:43,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:43,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:43,217 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:43,249 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=760 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/7734b3ae3dec4031a37dd01c02ef3867 2023-03-27 04:57:43,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=760 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/126eef92d78a4a7a822c9280fdba7620 2023-03-27 04:57:43,260 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/2bd2bbb38ad14ebbb5680bfd1adae94e as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/2bd2bbb38ad14ebbb5680bfd1adae94e 2023-03-27 04:57:43,263 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/2bd2bbb38ad14ebbb5680bfd1adae94e, entries=250, sequenceid=760, filesize=39.2 K 2023-03-27 04:57:43,264 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/7734b3ae3dec4031a37dd01c02ef3867 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/7734b3ae3dec4031a37dd01c02ef3867 2023-03-27 04:57:43,267 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/7734b3ae3dec4031a37dd01c02ef3867, entries=150, sequenceid=760, filesize=11.9 K 2023-03-27 04:57:43,267 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/126eef92d78a4a7a822c9280fdba7620 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/126eef92d78a4a7a822c9280fdba7620 2023-03-27 04:57:43,270 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/126eef92d78a4a7a822c9280fdba7620, entries=150, sequenceid=760, filesize=11.9 K 2023-03-27 04:57:43,270 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~174.43 KB/178620, heapSize ~482.77 KB/494352, currentSize=13.42 KB/13740 for 682db1751885437677f00c4ab8bf8ffc in 1249ms, sequenceid=760, compaction requested=true 2023-03-27 04:57:43,271 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:43,271 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:43,271 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-27 04:57:43,271 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-27 04:57:43,271 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:43,271 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:43,272 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137206 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-27 04:57:43,272 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 682db1751885437677f00c4ab8bf8ffc/A is initiating minor compaction (all files) 2023-03-27 04:57:43,272 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58426 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-27 04:57:43,272 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 682db1751885437677f00c4ab8bf8ffc/A in TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:43,272 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 682db1751885437677f00c4ab8bf8ffc/B is initiating minor compaction (all files) 2023-03-27 04:57:43,272 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/6414ceb30694420bb1931ea243db30e7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/999d816008c34fe7a21d4506fdba4f95, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/256c32ec99ae46b7be596d3e27581608, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/dac8264768564a3786e6776ae8510137, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/2bd2bbb38ad14ebbb5680bfd1adae94e] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp, totalSize=134.0 K 2023-03-27 04:57:43,272 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 682db1751885437677f00c4ab8bf8ffc/B in TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:43,272 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/2fcdaf36d5a14acc88836fd3e7f677da, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/15803a67f0324cfca3aa862d8a244787, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/0effd4bc255a403ca0be8f27e614e032, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/229f1a1679a34637ae3b21135e989419, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/7734b3ae3dec4031a37dd01c02ef3867] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp, totalSize=57.1 K 2023-03-27 04:57:43,272 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 6414ceb30694420bb1931ea243db30e7, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679893056341 2023-03-27 04:57:43,273 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 2fcdaf36d5a14acc88836fd3e7f677da, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679893056341 2023-03-27 04:57:43,273 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 999d816008c34fe7a21d4506fdba4f95, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679893057492 2023-03-27 04:57:43,273 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 15803a67f0324cfca3aa862d8a244787, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679893057492 2023-03-27 04:57:43,273 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 256c32ec99ae46b7be596d3e27581608, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1679893058604 2023-03-27 04:57:43,273 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 0effd4bc255a403ca0be8f27e614e032, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1679893058604 2023-03-27 04:57:43,273 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting dac8264768564a3786e6776ae8510137, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1679893059748 2023-03-27 04:57:43,273 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 229f1a1679a34637ae3b21135e989419, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1679893059748 2023-03-27 04:57:43,274 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 2bd2bbb38ad14ebbb5680bfd1adae94e, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1679893061913 2023-03-27 04:57:43,274 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 7734b3ae3dec4031a37dd01c02ef3867, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1679893061914 2023-03-27 04:57:43,281 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 682db1751885437677f00c4ab8bf8ffc#B#compaction#334 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:43,293 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 682db1751885437677f00c4ab8bf8ffc#A#compaction#335 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:43,317 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:43,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:43,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:43,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:43,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:43,323 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:43,323 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:43,323 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:43,323 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:43,323 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,323 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,324 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:43,324 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,324 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,324 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:43,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:43,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:43,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:43,325 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:43,325 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:43,325 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:43,330 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:43,330 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:43,330 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:43,330 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:43,330 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-27 04:57:43,330 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:43,330 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:43,331 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:43,331 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:43,331 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:43,331 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:43,334 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/f8643f7e208b41de9f6a8c83f97ec581 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/f8643f7e208b41de9f6a8c83f97ec581 2023-03-27 04:57:43,337 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 682db1751885437677f00c4ab8bf8ffc/A of 682db1751885437677f00c4ab8bf8ffc into f8643f7e208b41de9f6a8c83f97ec581(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:43,337 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:43,337 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc., storeName=682db1751885437677f00c4ab8bf8ffc/A, priority=11, startTime=1679893063271; duration=0sec 2023-03-27 04:57:43,337 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:43,337 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-27 04:57:43,339 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58426 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-27 04:57:43,339 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 682db1751885437677f00c4ab8bf8ffc/C is initiating minor compaction (all files) 2023-03-27 04:57:43,339 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 682db1751885437677f00c4ab8bf8ffc/C in TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:43,339 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/d400e690a98d4776b7805223b5bb40b7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/070690cfb599475cb5dc5f67a374f223, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/86c6ba31da93457c865cec7399c22a28, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/c3dca3502476411c99de162cbf80b433, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/126eef92d78a4a7a822c9280fdba7620] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp, totalSize=57.1 K 2023-03-27 04:57:43,339 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting d400e690a98d4776b7805223b5bb40b7, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679893056341 2023-03-27 04:57:43,339 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 070690cfb599475cb5dc5f67a374f223, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679893057492 2023-03-27 04:57:43,340 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 86c6ba31da93457c865cec7399c22a28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1679893058604 2023-03-27 04:57:43,340 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting c3dca3502476411c99de162cbf80b433, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1679893059748 2023-03-27 04:57:43,341 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 126eef92d78a4a7a822c9280fdba7620, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1679893061914 2023-03-27 04:57:43,352 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 682db1751885437677f00c4ab8bf8ffc#C#compaction#337 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:43,361 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/c5733df9c89f41159c4fb4fd411a24e3 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/c5733df9c89f41159c4fb4fd411a24e3 2023-03-27 04:57:43,364 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032713328d0b8f034ad096d42ce7988b062f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032713328d0b8f034ad096d42ce7988b062f 2023-03-27 04:57:43,365 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 682db1751885437677f00c4ab8bf8ffc/C of 682db1751885437677f00c4ab8bf8ffc into c5733df9c89f41159c4fb4fd411a24e3(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:43,365 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:43,365 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc., storeName=682db1751885437677f00c4ab8bf8ffc/C, priority=11, startTime=1679893063271; duration=0sec 2023-03-27 04:57:43,365 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:43,370 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=765, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/98f6d21eb43a493a9dedd8a005c5bbe1 2023-03-27 04:57:43,698 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/eaba31854ffe48219106afcb428f6bfe as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/eaba31854ffe48219106afcb428f6bfe 2023-03-27 04:57:43,701 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 682db1751885437677f00c4ab8bf8ffc/B of 682db1751885437677f00c4ab8bf8ffc into eaba31854ffe48219106afcb428f6bfe(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:43,701 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:43,701 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc., storeName=682db1751885437677f00c4ab8bf8ffc/B, priority=11, startTime=1679893063271; duration=0sec 2023-03-27 04:57:43,701 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:43,778 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=765 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/5f555a74cec84e75a4132da85c041bf1 2023-03-27 04:57:43,785 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=765 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/8545af0440654e9e9820dc4a73b91d5e 2023-03-27 04:57:43,788 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/98f6d21eb43a493a9dedd8a005c5bbe1 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/98f6d21eb43a493a9dedd8a005c5bbe1 2023-03-27 04:57:43,791 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/98f6d21eb43a493a9dedd8a005c5bbe1, entries=100, sequenceid=765, filesize=18.6 K 2023-03-27 04:57:43,792 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/5f555a74cec84e75a4132da85c041bf1 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/5f555a74cec84e75a4132da85c041bf1 2023-03-27 04:57:43,795 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/5f555a74cec84e75a4132da85c041bf1, entries=100, sequenceid=765, filesize=9.5 K 2023-03-27 04:57:43,796 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/8545af0440654e9e9820dc4a73b91d5e as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8545af0440654e9e9820dc4a73b91d5e 2023-03-27 04:57:43,798 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8545af0440654e9e9820dc4a73b91d5e, entries=100, sequenceid=765, filesize=9.5 K 2023-03-27 04:57:43,799 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 682db1751885437677f00c4ab8bf8ffc in 469ms, sequenceid=765, compaction requested=false 2023-03-27 04:57:43,799 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:43,799 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:43,799 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:43,799 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:43,799 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,799 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:43,799 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:43,800 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,800 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:43,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:43,800 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,800 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:43,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:43,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:43,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:43,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:43,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@330d458[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:43,802 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:43,802 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,802 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,802 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:43,802 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:43,802 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,802 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:43,802 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:43,803 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,803 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:43,803 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:43,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:43,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:43,803 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:43,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:43,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:43,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:43,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:43,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:43,805 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:43,805 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:43,805 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:43,805 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:43,805 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:43,806 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,806 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:43,806 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:43,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:43,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:43,807 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,809 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:43,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,810 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:43,810 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:43,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:43,810 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:43,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:43,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:43,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:43,810 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:43,810 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:43,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:43,811 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,811 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:43,811 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:43,811 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,811 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,811 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,811 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,911 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:43,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:43,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:43,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:43,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:43,914 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:43,914 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:43,914 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:43,914 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:43,914 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,914 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,915 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,915 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:43,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:43,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:43,915 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,915 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:43,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:43,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:43,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:43,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:43,916 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:43,916 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:43,916 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:43,916 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:43,916 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:43,916 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:43,917 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:43,917 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:43,917 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:43,917 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:43,917 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:43,917 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,917 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:43,917 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:43,918 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,918 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,918 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,918 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:43,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:43,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:43,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:43,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:43,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:43,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:43,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f574099[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:43,920 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:43,920 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,920 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,921 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:43,921 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:43,921 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:43,921 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,921 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:43,921 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,921 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:43,921 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:43,922 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:43,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:43,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:43,923 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:43,923 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:43,923 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:43,923 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:43,923 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:43,924 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,924 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:43,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:43,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,924 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:43,925 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:43,926 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:43,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:43,928 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,928 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:43,928 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:43,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:43,928 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:43,928 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:43,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:43,928 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:43,928 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:43,928 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:43,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:43,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:43,928 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:43,928 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,928 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:43,928 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:43,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:43,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:44,028 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:44,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:44,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:44,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:44,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:44,034 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:44,034 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:44,034 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:44,034 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:44,035 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:44,035 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:44,035 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:44,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:44,035 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:44,036 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:44,036 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:44,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:44,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:44,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:44,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:44,037 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:44,037 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:44,037 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:44,037 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:44,037 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:44,037 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:44,037 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:44,037 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:44,037 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:44,037 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:44,037 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:44,037 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,037 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:44,037 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:44,038 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,038 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:44,038 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:44,038 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:44,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:44,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:44,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:44,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4972b9ed[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:44,039 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:44,039 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:44,040 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:44,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:44,040 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:44,040 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:44,040 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:44,040 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,040 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:44,041 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,041 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:44,041 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:44,041 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:44,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:44,043 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:44,043 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:44,043 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:44,043 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:44,043 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:44,044 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:44,044 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:44,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:44,044 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:44,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:44,044 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:44,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:44,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:44,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:44,045 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:44,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:44,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,047 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:44,047 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,047 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:44,047 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:44,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:44,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:44,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:44,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:44,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:44,047 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:44,047 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:44,047 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:44,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:44,047 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:44,047 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,048 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:44,048 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:44,048 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:44,148 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:44,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:44,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:44,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:44,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:44,150 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:44,150 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:44,150 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:44,150 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:44,151 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:44,151 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:44,151 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:44,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:44,152 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:44,152 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,152 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:44,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:44,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:44,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:44,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:44,153 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:44,153 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:44,153 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:44,153 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:44,153 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:44,153 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:44,153 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:44,153 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-27 04:57:44,153 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:44,153 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:44,153 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:44,154 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:44,154 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:44,154 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:44,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:44,173 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:44,179 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303273d2d1827a84e4c0c9d87b20a64dd974a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303273d2d1827a84e4c0c9d87b20a64dd974a 2023-03-27 04:57:44,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679893064179, max timestamp=1679893064188Num uniques 859; 2023-03-27 04:57:44,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-27 04:57:44,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679893064179, max timestamp=1679893064188Num uniques 859; 2023-03-27 04:57:44,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-27 04:57:44,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679893064179, max timestamp=1679893064188Num uniques 859; 2023-03-27 04:57:44,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-27 04:57:44,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893124201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893124206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893124207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893124208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893124209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893124310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893124310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893124313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893124313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893124319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893124515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893124515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893124518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893124518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893124523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,583 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=779, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/a9f0e07be9b044f5aac78024b416b0ca 2023-03-27 04:57:44,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893124820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893124820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893124822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893124825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:44,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893124826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:44,991 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=779 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/063594a3e2a14c4f89f6d8c8736353bd 2023-03-27 04:57:45,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:45,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893125325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:45,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893125327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:45,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893125329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:45,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893125331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:45,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893125331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,400 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=779 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/13698f80b9b44c0289a8bb72dbaca19f 2023-03-27 04:57:45,403 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/a9f0e07be9b044f5aac78024b416b0ca as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/a9f0e07be9b044f5aac78024b416b0ca 2023-03-27 04:57:45,406 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/a9f0e07be9b044f5aac78024b416b0ca, entries=200, sequenceid=779, filesize=32.3 K 2023-03-27 04:57:45,407 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/063594a3e2a14c4f89f6d8c8736353bd as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/063594a3e2a14c4f89f6d8c8736353bd 2023-03-27 04:57:45,410 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/063594a3e2a14c4f89f6d8c8736353bd, entries=100, sequenceid=779, filesize=9.5 K 2023-03-27 04:57:45,411 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/13698f80b9b44c0289a8bb72dbaca19f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/13698f80b9b44c0289a8bb72dbaca19f 2023-03-27 04:57:45,415 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/13698f80b9b44c0289a8bb72dbaca19f, entries=100, sequenceid=779, filesize=9.5 K 2023-03-27 04:57:45,416 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 682db1751885437677f00c4ab8bf8ffc in 1263ms, sequenceid=779, compaction requested=true 2023-03-27 04:57:45,416 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:45,416 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:45,416 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:45,416 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:45,416 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,416 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:45,416 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:45,417 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:45,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:45,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:45,418 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:45,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:45,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:45,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:45,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:45,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:45,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a3dd4eb[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:45,420 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:45,420 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:45,421 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:45,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:45,421 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:45,421 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:45,421 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:45,421 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,421 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:45,422 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,422 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:45,422 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:45,422 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:45,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:45,424 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:45,424 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:45,424 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:45,424 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:45,424 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:45,425 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:45,425 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:45,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:45,425 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:45,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:45,426 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,427 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:45,428 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,428 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:45,428 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:45,428 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:45,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:45,428 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:45,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:45,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:45,428 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:45,429 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,429 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:45,429 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:45,429 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:45,429 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:45,429 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:45,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:45,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:45,529 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:45,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:45,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:45,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:45,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:45,532 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:45,532 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:45,532 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:45,532 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:45,532 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:45,532 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:45,533 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:45,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:45,533 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:45,533 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:45,533 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:45,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:45,534 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:45,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:45,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:45,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:45,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:45,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:45,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:45,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:45,534 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:45,534 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:45,534 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=147.60 KB heapSize=406.08 KB 2023-03-27 04:57:45,534 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:45,534 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:45,534 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:45,534 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:45,534 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:45,535 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:45,950 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303271b9df826dc3e476db13274ae207edcff to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303271b9df826dc3e476db13274ae207edcff 2023-03-27 04:57:46,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:46,338 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:46,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:46,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893126340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:46,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:46,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893126341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:46,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:46,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893126342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:46,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:46,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893126344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:46,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:46,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893126345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:46,355 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=814, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/2fcceb6a405647d1ac5bbc9529b36df5 2023-03-27 04:57:46,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:46,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893126446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:46,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:46,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893126447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:46,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:46,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893126447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:46,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:46,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893126451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:46,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:46,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893126650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:46,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:46,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893126653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:46,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:46,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893126653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:46,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:46,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893126655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:46,764 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=814 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/739a79c39f2649f69a05db6ad78c73a6 2023-03-27 04:57:46,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:46,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893126958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:46,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:46,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893126959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:46,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:46,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893126959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:46,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:46,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893126962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,175 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=814 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/c59414c5b92f4ba385a84bf38b2002ad 2023-03-27 04:57:47,179 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/2fcceb6a405647d1ac5bbc9529b36df5 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/2fcceb6a405647d1ac5bbc9529b36df5 2023-03-27 04:57:47,181 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/2fcceb6a405647d1ac5bbc9529b36df5, entries=150, sequenceid=814, filesize=25.4 K 2023-03-27 04:57:47,182 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/739a79c39f2649f69a05db6ad78c73a6 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/739a79c39f2649f69a05db6ad78c73a6 2023-03-27 04:57:47,185 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/739a79c39f2649f69a05db6ad78c73a6, entries=150, sequenceid=814, filesize=11.9 K 2023-03-27 04:57:47,186 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/c59414c5b92f4ba385a84bf38b2002ad as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/c59414c5b92f4ba385a84bf38b2002ad 2023-03-27 04:57:47,190 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/c59414c5b92f4ba385a84bf38b2002ad, entries=150, sequenceid=814, filesize=11.9 K 2023-03-27 04:57:47,191 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~406.03 KB/415776, currentSize=40.25 KB/41220 for 682db1751885437677f00c4ab8bf8ffc in 1657ms, sequenceid=814, compaction requested=true 2023-03-27 04:57:47,191 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:47,191 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:47,191 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:47,191 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:47,191 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,191 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:47,191 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:47,192 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,192 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:47,193 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:47,193 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:47,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:47,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7faa5de0[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:47,194 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:47,194 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:47,194 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:47,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:47,194 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:47,195 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:47,195 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,195 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:47,195 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:47,195 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,195 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:47,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:47,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:47,195 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:47,195 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:47,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:47,198 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:47,198 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:47,198 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:47,198 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:47,198 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:47,199 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:47,199 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:47,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:47,199 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:47,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:47,200 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:47,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:47,201 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,203 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:47,203 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:47,203 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:47,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:47,203 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:47,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:47,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:47,203 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:47,203 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:47,203 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,203 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:47,203 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:47,203 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:47,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:47,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:47,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:47,203 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:47,203 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:47,303 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:47,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:47,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:47,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:47,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:47,309 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:47,309 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:47,309 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:47,309 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:47,309 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:47,309 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:47,310 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:47,310 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:47,310 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,310 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:47,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:47,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:47,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:47,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:47,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:47,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:47,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:47,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:47,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:47,311 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:47,311 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:47,311 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-27 04:57:47,311 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:47,312 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:47,312 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:47,312 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:47,312 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:47,312 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:47,323 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327471fd2dc5708464e939d3a3453024faa to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327471fd2dc5708464e939d3a3453024faa 2023-03-27 04:57:47,327 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=823, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/a301c9e4211d45e5a4fb8a9853a443da 2023-03-27 04:57:47,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:47,466 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:47,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893067485, max timestamp=1679893067502Num uniques 859; 2023-03-27 04:57:47,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893067485, max timestamp=1679893067502Num uniques 859; 2023-03-27 04:57:47,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:57:47,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893067485, max timestamp=1679893067502Num uniques 859; 2023-03-27 04:57:47,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:57:47,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:57:47,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:47,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893127523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:47,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893127524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:47,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893127524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:47,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893127528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:47,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893127628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:47,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893127631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:47,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893127632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:47,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893127632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,734 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=823 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/3497c874be7648138fa73df46253ec52 2023-03-27 04:57:47,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:47,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893127836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:47,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893127836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:47,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893127836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:47,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:47,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893127837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,143 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=823 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/7fdf89c884cc48c6a9182b28212cd384 2023-03-27 04:57:48,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:48,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893128142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:48,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893128142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:48,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893128143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:48,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893128143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,146 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/a301c9e4211d45e5a4fb8a9853a443da as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/a301c9e4211d45e5a4fb8a9853a443da 2023-03-27 04:57:48,152 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/a301c9e4211d45e5a4fb8a9853a443da, entries=150, sequenceid=823, filesize=25.4 K 2023-03-27 04:57:48,152 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/3497c874be7648138fa73df46253ec52 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/3497c874be7648138fa73df46253ec52 2023-03-27 04:57:48,159 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/3497c874be7648138fa73df46253ec52, entries=150, sequenceid=823, filesize=11.9 K 2023-03-27 04:57:48,160 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/7fdf89c884cc48c6a9182b28212cd384 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/7fdf89c884cc48c6a9182b28212cd384 2023-03-27 04:57:48,163 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/7fdf89c884cc48c6a9182b28212cd384, entries=150, sequenceid=823, filesize=11.9 K 2023-03-27 04:57:48,163 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 682db1751885437677f00c4ab8bf8ffc in 852ms, sequenceid=823, compaction requested=true 2023-03-27 04:57:48,163 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:48,163 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:48,163 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:48,163 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:48,163 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,164 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:48,164 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:48,165 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,165 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:48,165 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:48,165 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:48,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:48,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a4c8904[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:48,167 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:48,167 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:48,167 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:48,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:48,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:48,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:48,167 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,167 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:48,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:48,168 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,168 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:48,168 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:48,168 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:48,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:48,170 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:48,170 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:48,170 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:48,170 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:48,170 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:48,171 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:48,171 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:48,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:48,171 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:48,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:48,171 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:48,173 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,174 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,174 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:48,174 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:48,174 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:48,174 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:48,174 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:48,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:48,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:48,174 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:48,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:48,174 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:48,174 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:48,174 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:48,174 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:48,174 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:48,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:48,274 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:48,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:48,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:48,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:48,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:48,277 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:48,277 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:48,277 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:48,277 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:48,277 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:48,277 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:48,278 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,278 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:48,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:48,279 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,279 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:48,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:48,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:48,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:48,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:48,279 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:48,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:48,279 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:48,279 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:48,279 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:48,279 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:48,279 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:48,279 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:48,280 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:48,280 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-27 04:57:48,280 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:48,280 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:48,280 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:48,280 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:48,280 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:48,280 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:48,290 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327f2f5c6b7888d420ea6949822542365f5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327f2f5c6b7888d420ea6949822542365f5 2023-03-27 04:57:48,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:48,359 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:48,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:48,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893128393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:48,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893128500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:48,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893128645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:48,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893128646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:48,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893128646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:48,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893128649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:48,695 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=860, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/adb7ad8b4493425ea5300bc1369d558c 2023-03-27 04:57:48,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:48,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893128706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:49,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893129012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,111 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=860 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/7adc2bcf4adf4270892e7af81b2650fb 2023-03-27 04:57:49,519 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=860 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/63d99e24251f4a1f9ca024bf2522ee66 2023-03-27 04:57:49,522 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/adb7ad8b4493425ea5300bc1369d558c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/adb7ad8b4493425ea5300bc1369d558c 2023-03-27 04:57:49,525 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/adb7ad8b4493425ea5300bc1369d558c, entries=150, sequenceid=860, filesize=25.4 K 2023-03-27 04:57:49,526 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/7adc2bcf4adf4270892e7af81b2650fb as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/7adc2bcf4adf4270892e7af81b2650fb 2023-03-27 04:57:49,529 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/7adc2bcf4adf4270892e7af81b2650fb, entries=150, sequenceid=860, filesize=11.9 K 2023-03-27 04:57:49,529 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/63d99e24251f4a1f9ca024bf2522ee66 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/63d99e24251f4a1f9ca024bf2522ee66 2023-03-27 04:57:49,531 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/63d99e24251f4a1f9ca024bf2522ee66, entries=150, sequenceid=860, filesize=11.9 K 2023-03-27 04:57:49,532 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 682db1751885437677f00c4ab8bf8ffc in 1252ms, sequenceid=860, compaction requested=true 2023-03-27 04:57:49,532 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:49,532 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:49,532 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:49,532 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:49,532 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,532 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:49,532 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:49,533 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,533 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:49,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:49,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:49,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:49,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16adfc71[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:49,535 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:49,535 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,535 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,535 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:49,535 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:49,535 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:49,536 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,536 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:49,536 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,536 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:49,536 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:49,536 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:49,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:49,538 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:49,538 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:49,538 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:49,538 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:49,538 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:49,539 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,539 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,539 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:49,539 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:49,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,540 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,542 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:49,542 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:49,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:49,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:49,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:49,542 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,542 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,542 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,542 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,542 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,542 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:49,542 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:49,542 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:49,542 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:49,542 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:49,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:49,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:49,548 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:57:49,548 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:49,549 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:49,549 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:49,549 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:49,549 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:49,549 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:49,642 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:49,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:49,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:49,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:49,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:49,645 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:49,645 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:49,645 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:49,645 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:49,646 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,646 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,646 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,646 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:49,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:49,646 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,646 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:49,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:49,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:49,647 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:49,647 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:49,648 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:49,648 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:49,648 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:49,648 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:49,648 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:49,648 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:49,648 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:49,648 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:49,648 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:49,648 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,648 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:49,648 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:49,649 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,649 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:49,649 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,649 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:49,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:49,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a3f8c3[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:49,650 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:49,650 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,651 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:49,651 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:49,651 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:49,651 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,651 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:49,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:49,651 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:49,652 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:49,652 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:49,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:49,653 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:49,653 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:49,653 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:49,653 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:49,653 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:49,654 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,654 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,654 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:49,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:49,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:49,654 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893069617, max timestamp=1679893069654Num uniques 859; 2023-03-27 04:57:49,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:57:49,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893069617, max timestamp=1679893069654Num uniques 859; 2023-03-27 04:57:49,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893069617, max timestamp=1679893069654Num uniques 859; 2023-03-27 04:57:49,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:57:49,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:57:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,657 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,659 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,659 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,659 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:49,659 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:49,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:49,659 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:49,659 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:49,660 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:49,660 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:49,659 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:49,659 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:49,660 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,660 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:49,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893129682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679893069681, max timestamp=1679893069683Num uniques 1018; 2023-03-27 04:57:49,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-27 04:57:49,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679893069681, max timestamp=1679893069683Num uniques 1018; 2023-03-27 04:57:49,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679893069681, max timestamp=1679893069683Num uniques 1018; 2023-03-27 04:57:49,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-27 04:57:49,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-27 04:57:49,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:49,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893129714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:49,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893129714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:49,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893129715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:49,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893129716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,760 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:49,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:49,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:49,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:49,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:49,762 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:49,762 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:49,762 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:49,762 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:49,763 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,763 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,763 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,763 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:49,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:49,764 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,764 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:49,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:49,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:49,764 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:49,764 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:49,764 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:49,764 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:49,765 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:49,765 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:49,765 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:49,765 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:49,765 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:49,765 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:49,765 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:49,765 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,765 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:49,765 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:49,765 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:49,765 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:49,766 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,766 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:49,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:49,767 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:49,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fcbe02a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:49,767 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,768 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:49,768 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:49,768 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:49,768 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,768 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:49,769 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,769 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:49,769 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:49,769 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:49,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:49,771 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:49,771 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:49,771 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:49,771 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:49,771 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:49,772 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,772 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,772 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:49,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,772 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:49,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,773 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,776 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:49,776 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:49,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:49,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:49,776 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,776 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,776 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,776 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,776 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:49,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:49,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:49,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:49,776 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,777 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,777 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,777 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:49,777 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:49,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:49,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893129784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:49,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34598 deadline: 1679893129822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:49,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893129822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:49,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893129823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:49,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34566 deadline: 1679893129823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,877 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:49,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:49,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:49,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:49,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:49,880 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:49,880 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:49,880 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:49,880 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:49,880 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,880 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,881 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,881 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:49,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:49,881 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,881 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:49,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:49,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:49,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:49,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:49,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:49,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:49,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:49,882 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:49,882 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:49,882 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:49,882 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:49,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:49,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:49,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:49,882 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:49,883 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,883 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:49,883 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,883 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:49,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:49,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@161487e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:49,884 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:49,884 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,885 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,885 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:49,885 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:49,885 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,885 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:49,885 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:49,886 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,886 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:49,886 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:49,886 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:49,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:49,887 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:49,887 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:49,887 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:49,887 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:49,887 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:49,888 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,888 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:49,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,888 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:49,888 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:49,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:49,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:49,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:49,891 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:49,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:49,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:49,891 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:49,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,891 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:49,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,891 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:49,957 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327d0681f7078ff4fdaa1734a2b3615463a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327d0681f7078ff4fdaa1734a2b3615463a 2023-03-27 04:57:49,991 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:49,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:49,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:49,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:49,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:49,994 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:49,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:49,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:49,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:49,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,995 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:49,995 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:49,995 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,995 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:49,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:49,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:49,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:49,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:49,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:49,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:49,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:49,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:49,996 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:49,996 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:49,996 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:49,996 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:49,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:49,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:49,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:49,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:49,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893129991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:49,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,997 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:49,997 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,998 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:49,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:49,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78d66257[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:49,998 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:49,999 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,999 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:49,999 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:49,999 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:49,999 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:49,999 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:49,999 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:50,000 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,000 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:50,000 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,000 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:50,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:50,001 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:50,001 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:50,002 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:50,002 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:50,002 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:50,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,003 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,004 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,005 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,005 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:50,005 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,005 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:50,005 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:50,006 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,006 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,006 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:50,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:50,006 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,006 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,006 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:50,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:50,006 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:50,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:50,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34598 deadline: 1679893130035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:50,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:50,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893130035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893130035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:50,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34566 deadline: 1679893130035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,106 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:50,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:50,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:50,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:50,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:50,109 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:50,109 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:50,109 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:50,109 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:50,109 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,109 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,110 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,110 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:50,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:50,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:50,110 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,110 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:50,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:50,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:50,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:50,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:50,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:50,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:50,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:50,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:50,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:50,111 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:50,111 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:50,111 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:50,111 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:50,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:50,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:50,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:50,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:50,112 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,112 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,112 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,112 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:50,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65aee86e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:50,114 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:50,114 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,114 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,114 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:50,114 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:50,114 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,114 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:50,114 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:50,116 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:50,116 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,116 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,116 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:50,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:50,118 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:50,118 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:50,118 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:50,118 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:50,118 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:50,119 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,119 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,119 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,119 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,120 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,122 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:50,122 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,122 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,122 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,122 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,122 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,122 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:50,122 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,122 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:50,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:50,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:50,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,122 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:50,122 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:50,222 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:50,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:50,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:50,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:50,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:50,225 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:50,225 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:50,225 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:50,225 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:50,225 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,225 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,226 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:50,226 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:50,226 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,226 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:50,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:50,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:50,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:50,227 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:50,227 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:50,227 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:50,227 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:50,228 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:50,228 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:50,228 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:50,228 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:50,228 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:50,228 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:50,228 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:50,228 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,228 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:50,228 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:50,229 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,229 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,229 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,229 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:50,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15149e41[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:50,230 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:50,230 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,231 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,231 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:50,231 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,231 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:50,231 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:50,231 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:50,231 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,231 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:50,232 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,232 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:50,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:50,233 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:50,233 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:50,233 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:50,233 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:50,233 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:50,234 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,234 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,234 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,234 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,235 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,236 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,236 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:50,236 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,236 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:50,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:50,236 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:50,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:50,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:50,237 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:50,236 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,237 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:50,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,237 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,237 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,237 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,237 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:50,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893130298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,337 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:50,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:50,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:50,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:50,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:50,339 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:50,339 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:50,339 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:50,339 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:50,339 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,340 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,340 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:50,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:50,350 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,350 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:50,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:50,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:50,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:50,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:50,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:50,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:50,352 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:50,352 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:50,352 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:50,352 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:50,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:50,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:50,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:50,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:50,353 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:50,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ebd0625[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:50,354 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:50,354 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,355 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,355 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:50,355 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:50,355 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:50,355 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,355 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:50,355 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,355 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:50,355 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,355 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:50,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893130351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,357 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:50,357 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:50,357 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:50,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:50,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:50,357 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:50,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34598 deadline: 1679893130351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,357 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:50,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:50,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893130351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:50,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34566 deadline: 1679893130351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,359 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,360 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,361 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,361 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,361 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:50,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:50,361 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:50,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:50,361 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,361 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:50,361 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,361 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:50,361 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:50,361 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,361 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,361 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,362 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=870, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/4107b42c00b74571918c8d9c6f0c2da7 2023-03-27 04:57:50,461 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:50,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:50,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:50,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:50,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:50,464 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:50,464 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:50,464 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:50,464 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:50,464 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,464 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,465 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,465 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:50,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:50,465 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,465 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:50,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:50,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:50,466 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:50,466 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:50,466 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:50,466 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:50,466 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:50,466 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:50,466 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:50,467 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:50,467 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:50,467 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:50,467 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:50,467 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,467 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:50,467 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:50,467 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,467 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,468 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,468 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:50,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55b20b7f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:50,469 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:50,469 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,470 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,470 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:50,470 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:50,470 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:50,470 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,470 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:50,470 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:50,470 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,470 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,471 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:50,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:50,472 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:50,472 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:50,472 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:50,472 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:50,472 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:50,473 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,473 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,473 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,473 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,474 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,476 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,476 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:50,476 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,476 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:50,476 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,476 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:50,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:50,476 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,476 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,477 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,477 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:50,477 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:50,477 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,577 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:50,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:50,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:50,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:50,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:50,580 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:50,580 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:50,580 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:50,580 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:50,580 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,580 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,581 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:50,581 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:50,581 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,581 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:50,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:50,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:50,582 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:50,583 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:50,583 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:50,583 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:50,583 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:50,583 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:50,583 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:50,583 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:50,583 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:50,583 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:50,583 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:50,583 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,583 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:50,583 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:50,584 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,584 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,584 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,584 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:50,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@474d743f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:50,585 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:50,585 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:50,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:50,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:50,586 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,586 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:50,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:50,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,587 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:50,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:50,589 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:50,589 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:50,589 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:50,589 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:50,589 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:50,590 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,590 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,590 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,590 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,591 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,593 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:50,593 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,593 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:50,594 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,594 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:50,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:50,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:50,594 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,594 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,594 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:50,594 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,594 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:50,594 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,594 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,694 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:50,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:50,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:50,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:50,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:50,697 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:50,697 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:50,697 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:50,697 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:50,697 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,697 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,698 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:50,698 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:50,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:50,698 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,698 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:50,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:50,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:50,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:50,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:50,700 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:50,700 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:50,701 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:50,701 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:50,701 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:50,701 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:50,701 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:50,701 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:50,701 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:50,701 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:50,701 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:50,701 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,701 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:50,701 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:50,702 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,702 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,702 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,702 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:50,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@761a0c01[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:50,704 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:50,704 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,704 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:50,704 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,705 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:50,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:50,705 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:50,705 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:50,705 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,705 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,705 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:50,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:50,707 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:50,707 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:50,707 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:50,707 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:50,707 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:50,709 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,709 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,709 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,709 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,710 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:50,712 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,712 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:50,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:50,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:50,712 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,712 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,712 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:50,712 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,712 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:50,712 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,712 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:50,712 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,712 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,770 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=870 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/ad8480e05bbe46f99f7a1502aa1086f1 2023-03-27 04:57:50,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:50,812 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:50,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34586 deadline: 1679893130807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:50,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:50,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:50,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:50,814 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:50,814 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:50,814 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:50,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:50,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,815 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:50,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:50,816 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,816 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:50,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:50,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:50,817 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:50,817 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:50,817 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:50,818 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:50,818 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:50,818 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:50,818 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:50,818 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:50,818 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:50,818 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:50,818 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:50,818 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,818 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:50,818 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:50,819 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,819 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,819 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,819 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:50,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e0b0c9[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:50,820 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:50,820 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,821 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,821 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,821 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:50,821 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:50,821 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:50,821 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:50,822 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,822 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,822 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,822 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:50,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:50,825 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:50,825 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:50,825 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:50,825 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:50,825 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:50,825 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,825 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,825 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,826 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,827 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,828 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,828 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,828 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:50,829 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:50,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:50,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:50,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:50,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:50,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,829 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:50,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,829 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:50,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:50,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34598 deadline: 1679893130859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:50,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34588 deadline: 1679893130859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:50,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34566 deadline: 1679893130861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:50,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34614 deadline: 1679893130861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,929 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:50,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:50,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:50,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:50,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:50,932 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:50,932 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:50,932 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:50,932 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:50,932 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,932 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,933 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,933 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:50,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:50,933 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,933 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:50,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:50,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:50,934 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:50,934 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:50,934 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:50,934 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:50,934 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:50,934 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:50,934 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:50,934 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:50,934 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:50,935 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:50,935 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:50,935 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,935 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:50,935 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:50,935 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,935 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,936 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,936 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:50,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ccba59[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:50,937 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:50,937 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,937 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:50,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:50,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:50,938 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,938 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:50,938 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,938 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,938 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,938 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:50,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:50,940 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:50,940 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:50,940 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:50,940 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:50,940 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:50,941 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,941 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,941 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:50,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,941 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,942 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,943 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:50,943 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:50,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:50,944 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,943 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:50,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:50,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:50,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:50,944 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:50,944 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:50,944 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:50,944 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:50,944 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:50,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:50,944 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:50,944 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:50,944 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,044 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:51,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:51,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:51,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:51,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:51,047 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:51,047 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:51,047 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:51,047 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:51,047 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,047 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:51,047 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,047 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:51,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:51,048 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,048 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:51,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:51,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:51,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:51,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:51,049 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:51,049 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:51,049 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:51,049 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:51,049 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:51,049 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:51,049 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:51,049 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:51,049 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:51,049 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:51,049 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:51,049 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,049 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:51,049 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:51,050 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,050 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:51,050 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:51,050 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:51,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:51,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@690cd253[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:51,051 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:51,051 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:51,052 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:51,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:51,052 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:51,052 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:51,052 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:51,052 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,052 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:51,053 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:51,053 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:51,053 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:51,053 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:51,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:51,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:51,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:51,054 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:51,055 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:51,055 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:51,055 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:51,055 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:51,055 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,055 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:51,055 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:51,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,056 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:51,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:51,057 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,058 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,058 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:51,058 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:51,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:51,058 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:51,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:51,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:51,058 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:51,058 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:51,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:51,058 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:51,058 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:51,058 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:51,058 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:51,058 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:51,059 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:51,159 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:51,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:51,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:51,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:51,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:51,161 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:51,161 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:51,161 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:51,161 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:51,162 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,162 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:51,162 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:51,162 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:51,162 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,162 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:51,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:51,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:51,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:51,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:51,163 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:51,163 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:51,163 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:51,164 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:51,164 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:51,164 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:51,164 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. started... 2023-03-27 04:57:51,164 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. as already flushing 2023-03-27 04:57:51,164 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:51,164 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:51,164 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:51,164 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,164 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:51,164 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:51,165 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:51,165 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:51,165 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:51,165 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:51,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:51,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f50a177[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:51,166 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:51,166 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:51,166 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:51,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:51,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:51,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:51,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:51,167 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,167 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:51,167 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:51,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:51,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:51,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:51,169 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:51,169 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:51,169 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:51,169 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:51,169 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:51,170 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:51,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:51,170 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,171 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:51,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:51,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:51,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:51,172 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:51,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,173 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,175 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:51,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:51,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:51,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:51,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:51,176 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:51,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:51,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:51,176 DEBUG [Thread-2304] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:51,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:51,176 DEBUG [Thread-2304] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:51,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:51,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:51,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:51,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:51,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:51,178 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=870 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/b0020610f9a14e9ea255572d12e0d761 2023-03-27 04:57:51,180 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/4107b42c00b74571918c8d9c6f0c2da7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/4107b42c00b74571918c8d9c6f0c2da7 2023-03-27 04:57:51,183 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/4107b42c00b74571918c8d9c6f0c2da7, entries=150, sequenceid=870, filesize=25.4 K 2023-03-27 04:57:51,183 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/ad8480e05bbe46f99f7a1502aa1086f1 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/ad8480e05bbe46f99f7a1502aa1086f1 2023-03-27 04:57:51,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/ad8480e05bbe46f99f7a1502aa1086f1, entries=100, sequenceid=870, filesize=9.5 K 2023-03-27 04:57:51,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/b0020610f9a14e9ea255572d12e0d761 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/b0020610f9a14e9ea255572d12e0d761 2023-03-27 04:57:51,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/b0020610f9a14e9ea255572d12e0d761, entries=100, sequenceid=870, filesize=9.5 K 2023-03-27 04:57:51,189 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 682db1751885437677f00c4ab8bf8ffc in 1641ms, sequenceid=870, compaction requested=true 2023-03-27 04:57:51,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:51,189 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:51,189 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-27 04:57:51,189 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-27 04:57:51,189 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:51,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:51,191 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 182716 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-27 04:57:51,191 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 682db1751885437677f00c4ab8bf8ffc/A is initiating minor compaction (all files) 2023-03-27 04:57:51,191 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 78380 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-27 04:57:51,191 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 682db1751885437677f00c4ab8bf8ffc/A in TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:51,191 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 682db1751885437677f00c4ab8bf8ffc/B is initiating minor compaction (all files) 2023-03-27 04:57:51,191 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/f8643f7e208b41de9f6a8c83f97ec581, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/98f6d21eb43a493a9dedd8a005c5bbe1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/a9f0e07be9b044f5aac78024b416b0ca, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/2fcceb6a405647d1ac5bbc9529b36df5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/a301c9e4211d45e5a4fb8a9853a443da, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/adb7ad8b4493425ea5300bc1369d558c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/4107b42c00b74571918c8d9c6f0c2da7] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp, totalSize=178.4 K 2023-03-27 04:57:51,191 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 682db1751885437677f00c4ab8bf8ffc/B in TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:51,191 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/eaba31854ffe48219106afcb428f6bfe, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/5f555a74cec84e75a4132da85c041bf1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/063594a3e2a14c4f89f6d8c8736353bd, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/739a79c39f2649f69a05db6ad78c73a6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/3497c874be7648138fa73df46253ec52, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/7adc2bcf4adf4270892e7af81b2650fb, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/ad8480e05bbe46f99f7a1502aa1086f1] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp, totalSize=76.5 K 2023-03-27 04:57:51,191 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting f8643f7e208b41de9f6a8c83f97ec581, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1679893061914 2023-03-27 04:57:51,192 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting eaba31854ffe48219106afcb428f6bfe, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1679893061914 2023-03-27 04:57:51,192 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 5f555a74cec84e75a4132da85c041bf1, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=765, earliestPutTs=1679893062029 2023-03-27 04:57:51,192 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 98f6d21eb43a493a9dedd8a005c5bbe1, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=765, earliestPutTs=1679893062029 2023-03-27 04:57:51,192 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 063594a3e2a14c4f89f6d8c8736353bd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=779, earliestPutTs=1679893064115 2023-03-27 04:57:51,192 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting a9f0e07be9b044f5aac78024b416b0ca, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=779, earliestPutTs=1679893064115 2023-03-27 04:57:51,192 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 739a79c39f2649f69a05db6ad78c73a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=814, earliestPutTs=1679893064205 2023-03-27 04:57:51,192 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 2fcceb6a405647d1ac5bbc9529b36df5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=814, earliestPutTs=1679893064205 2023-03-27 04:57:51,193 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 3497c874be7648138fa73df46253ec52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=823, earliestPutTs=1679893066340 2023-03-27 04:57:51,193 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting a301c9e4211d45e5a4fb8a9853a443da, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=823, earliestPutTs=1679893066340 2023-03-27 04:57:51,193 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 7adc2bcf4adf4270892e7af81b2650fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=860, earliestPutTs=1679893067520 2023-03-27 04:57:51,193 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting adb7ad8b4493425ea5300bc1369d558c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=860, earliestPutTs=1679893067520 2023-03-27 04:57:51,193 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting ad8480e05bbe46f99f7a1502aa1086f1, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=870, earliestPutTs=1679893068392 2023-03-27 04:57:51,193 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 4107b42c00b74571918c8d9c6f0c2da7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=870, earliestPutTs=1679893068384 2023-03-27 04:57:51,200 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 682db1751885437677f00c4ab8bf8ffc#B#compaction#355 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:51,201 DEBUG [Thread-2305] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x47e30d1c to 127.0.0.1:61059 2023-03-27 04:57:51,201 DEBUG [Thread-2305] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:51,204 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 682db1751885437677f00c4ab8bf8ffc#A#compaction#356 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:51,204 DEBUG [Thread-2309] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ded66ad to 127.0.0.1:61059 2023-03-27 04:57:51,204 DEBUG [Thread-2309] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:51,206 DEBUG [Thread-2313] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x228ff1a4 to 127.0.0.1:61059 2023-03-27 04:57:51,206 DEBUG [Thread-2313] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:51,211 DEBUG [Thread-2311] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14d061d2 to 127.0.0.1:61059 2023-03-27 04:57:51,211 DEBUG [Thread-2311] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:51,213 DEBUG [Thread-2307] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36a9ece3 to 127.0.0.1:61059 2023-03-27 04:57:51,214 DEBUG [Thread-2307] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:51,276 DEBUG [Thread-2304] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:51,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:51,618 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/88183f65fa5c44ed80ba63e9dc95b22d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/88183f65fa5c44ed80ba63e9dc95b22d 2023-03-27 04:57:51,621 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 682db1751885437677f00c4ab8bf8ffc/B of 682db1751885437677f00c4ab8bf8ffc into 88183f65fa5c44ed80ba63e9dc95b22d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:51,621 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:51,621 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc., storeName=682db1751885437677f00c4ab8bf8ffc/B, priority=9, startTime=1679893071189; duration=0sec 2023-03-27 04:57:51,621 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:57:51,621 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-27 04:57:51,623 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 78380 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-27 04:57:51,623 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 682db1751885437677f00c4ab8bf8ffc/C is initiating minor compaction (all files) 2023-03-27 04:57:51,623 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 682db1751885437677f00c4ab8bf8ffc/C in TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:51,623 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/c5733df9c89f41159c4fb4fd411a24e3, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8545af0440654e9e9820dc4a73b91d5e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/13698f80b9b44c0289a8bb72dbaca19f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/c59414c5b92f4ba385a84bf38b2002ad, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/7fdf89c884cc48c6a9182b28212cd384, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/63d99e24251f4a1f9ca024bf2522ee66, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/b0020610f9a14e9ea255572d12e0d761] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp, totalSize=76.5 K 2023-03-27 04:57:51,623 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting c5733df9c89f41159c4fb4fd411a24e3, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1679893061914 2023-03-27 04:57:51,624 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 8545af0440654e9e9820dc4a73b91d5e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=765, earliestPutTs=1679893062029 2023-03-27 04:57:51,624 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 13698f80b9b44c0289a8bb72dbaca19f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=779, earliestPutTs=1679893064115 2023-03-27 04:57:51,624 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting c59414c5b92f4ba385a84bf38b2002ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=814, earliestPutTs=1679893064205 2023-03-27 04:57:51,624 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 7fdf89c884cc48c6a9182b28212cd384, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=823, earliestPutTs=1679893066340 2023-03-27 04:57:51,625 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 63d99e24251f4a1f9ca024bf2522ee66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=860, earliestPutTs=1679893067520 2023-03-27 04:57:51,625 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting b0020610f9a14e9ea255572d12e0d761, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=870, earliestPutTs=1679893068392 2023-03-27 04:57:51,635 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 682db1751885437677f00c4ab8bf8ffc#C#compaction#357 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 65.38 MB/second 2023-03-27 04:57:51,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:51,825 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=140.89 KB heapSize=399.38 KB 2023-03-27 04:57:51,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:51,825 DEBUG [Thread-2300] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5da5030e to 127.0.0.1:61059 2023-03-27 04:57:51,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:51,825 DEBUG [Thread-2300] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:51,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:51,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:51,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:51,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:57:51,869 DEBUG [Thread-2296] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x74a355de to 127.0.0.1:61059 2023-03-27 04:57:51,869 DEBUG [Thread-2294] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d243e46 to 127.0.0.1:61059 2023-03-27 04:57:51,869 DEBUG [Thread-2296] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:51,869 DEBUG [Thread-2294] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:51,869 DEBUG [Thread-2302] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x74d6bf18 to 127.0.0.1:61059 2023-03-27 04:57:51,869 DEBUG [Thread-2302] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:51,873 DEBUG [Thread-2298] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x57b6628e to 127.0.0.1:61059 2023-03-27 04:57:51,873 DEBUG [Thread-2298] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:51,873 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-27 04:57:51,873 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 60 2023-03-27 04:57:51,873 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 292 2023-03-27 04:57:51,873 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 59 2023-03-27 04:57:51,873 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 203 2023-03-27 04:57:51,873 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 246 2023-03-27 04:57:51,873 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-27 04:57:51,874 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-27 04:57:51,874 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(402): scanned 1905 2023-03-27 04:57:51,874 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(403): verified 5715 rows 2023-03-27 04:57:51,874 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(402): scanned 1903 2023-03-27 04:57:51,874 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(403): verified 5709 rows 2023-03-27 04:57:51,874 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(402): scanned 1892 2023-03-27 04:57:51,874 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(403): verified 5676 rows 2023-03-27 04:57:51,874 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(402): scanned 1898 2023-03-27 04:57:51,874 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(403): verified 5694 rows 2023-03-27 04:57:51,874 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(402): scanned 1906 2023-03-27 04:57:51,874 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(403): verified 5718 rows 2023-03-27 04:57:51,874 INFO [Listener at localhost.localdomain/43311] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-27 04:57:51,874 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1bd9eb59 to 127.0.0.1:61059 2023-03-27 04:57:51,875 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:57:51,876 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-27 04:57:51,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-27 04:57:51,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:51,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-27 04:57:51,881 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893071881"}]},"ts":"1679893071881"} 2023-03-27 04:57:51,882 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-27 04:57:51,892 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-27 04:57:51,893 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=682db1751885437677f00c4ab8bf8ffc, UNASSIGN}] 2023-03-27 04:57:51,893 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=682db1751885437677f00c4ab8bf8ffc, UNASSIGN 2023-03-27 04:57:51,894 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=682db1751885437677f00c4ab8bf8ffc, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:51,894 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893071894"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679893071894"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679893071894"}]},"ts":"1679893071894"} 2023-03-27 04:57:51,895 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:57:51,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-27 04:57:52,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:52,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 682db1751885437677f00c4ab8bf8ffc, disabling compactions & flushes 2023-03-27 04:57:52,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 2 compactions & cache flush to complete for region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:52,049 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/082b1b6c56ed44a6b3e055b26cb65fc9 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/082b1b6c56ed44a6b3e055b26cb65fc9 2023-03-27 04:57:52,056 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 682db1751885437677f00c4ab8bf8ffc/C of 682db1751885437677f00c4ab8bf8ffc into 082b1b6c56ed44a6b3e055b26cb65fc9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:57:52,056 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:52,056 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc., storeName=682db1751885437677f00c4ab8bf8ffc/C, priority=9, startTime=1679893071190; duration=0sec 2023-03-27 04:57:52,056 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:52,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-27 04:57:52,246 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327c427972338bb41749226ab35ad222d1c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327c427972338bb41749226ab35ad222d1c 2023-03-27 04:57:52,438 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/7c742b9df205427e839215be98a80b2e as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/7c742b9df205427e839215be98a80b2e 2023-03-27 04:57:52,445 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 682db1751885437677f00c4ab8bf8ffc/A of 682db1751885437677f00c4ab8bf8ffc into 7c742b9df205427e839215be98a80b2e(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-27 04:57:52,445 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:52,445 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:52,445 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc., storeName=682db1751885437677f00c4ab8bf8ffc/A, priority=9, startTime=1679893071189; duration=1sec 2023-03-27 04:57:52,445 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:57:52,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-27 04:57:52,653 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=925, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/1ab57e58917e4307a53d3662820bf5b4 2023-03-27 04:57:52,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-27 04:57:53,068 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=925 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/ee33f77cf0a544189cd40c5342088e2b 2023-03-27 04:57:53,484 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=925 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/af8c40fd811b4d0290b5bbaec93bba30 2023-03-27 04:57:53,489 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/1ab57e58917e4307a53d3662820bf5b4 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/1ab57e58917e4307a53d3662820bf5b4 2023-03-27 04:57:53,494 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/1ab57e58917e4307a53d3662820bf5b4, entries=150, sequenceid=925, filesize=25.4 K 2023-03-27 04:57:53,495 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/ee33f77cf0a544189cd40c5342088e2b as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/ee33f77cf0a544189cd40c5342088e2b 2023-03-27 04:57:53,501 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/ee33f77cf0a544189cd40c5342088e2b, entries=150, sequenceid=925, filesize=11.9 K 2023-03-27 04:57:53,502 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/af8c40fd811b4d0290b5bbaec93bba30 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/af8c40fd811b4d0290b5bbaec93bba30 2023-03-27 04:57:53,507 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/af8c40fd811b4d0290b5bbaec93bba30, entries=150, sequenceid=925, filesize=11.9 K 2023-03-27 04:57:53,508 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~399.33 KB/408912, currentSize=26.84 KB/27480 for 682db1751885437677f00c4ab8bf8ffc in 1683ms, sequenceid=925, compaction requested=false 2023-03-27 04:57:53,508 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:53,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:53,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:53,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. after waiting 0 ms 2023-03-27 04:57:53,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:53,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 682db1751885437677f00c4ab8bf8ffc 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-27 04:57:53,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=A 2023-03-27 04:57:53,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:53,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=B 2023-03-27 04:57:53,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:53,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 682db1751885437677f00c4ab8bf8ffc, store=C 2023-03-27 04:57:53,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:53,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303277780152d4beb4b5b9842359b59beeac6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303277780152d4beb4b5b9842359b59beeac6 2023-03-27 04:57:53,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-27 04:57:54,336 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=934, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/ed7e000b27064395ae023482de51f5ff 2023-03-27 04:57:54,754 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=934 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/f8a71c81730c4cec815335991606f2b2 2023-03-27 04:57:55,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=934 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/8cf50da2f701431fb180e7e83379e982 2023-03-27 04:57:55,180 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/A/ed7e000b27064395ae023482de51f5ff as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/ed7e000b27064395ae023482de51f5ff 2023-03-27 04:57:55,185 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/ed7e000b27064395ae023482de51f5ff, entries=100, sequenceid=934, filesize=18.6 K 2023-03-27 04:57:55,186 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/B/f8a71c81730c4cec815335991606f2b2 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/f8a71c81730c4cec815335991606f2b2 2023-03-27 04:57:55,192 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/f8a71c81730c4cec815335991606f2b2, entries=100, sequenceid=934, filesize=9.5 K 2023-03-27 04:57:55,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/.tmp/C/8cf50da2f701431fb180e7e83379e982 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8cf50da2f701431fb180e7e83379e982 2023-03-27 04:57:55,200 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8cf50da2f701431fb180e7e83379e982, entries=100, sequenceid=934, filesize=9.5 K 2023-03-27 04:57:55,201 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 682db1751885437677f00c4ab8bf8ffc in 1693ms, sequenceid=934, compaction requested=true 2023-03-27 04:57:55,224 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/50bab4d4ff8e49eba9ffa453f9e25522, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/76eb603b9a8e4b4791dcf4760e99b78f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/f801182b8e4b4ca9b33b4181ea6a91df, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/218903ba93c84d238678c09ada4bdf47, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/f8b64f2bbe704392b8ba5182f11c03ff, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/77d099bab9ff42fbaff40ec9ec172aed, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/11469f5721004b489903e1bc8616fee5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/6414ceb30694420bb1931ea243db30e7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/999d816008c34fe7a21d4506fdba4f95, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/256c32ec99ae46b7be596d3e27581608, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/dac8264768564a3786e6776ae8510137, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/2bd2bbb38ad14ebbb5680bfd1adae94e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/f8643f7e208b41de9f6a8c83f97ec581, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/98f6d21eb43a493a9dedd8a005c5bbe1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/a9f0e07be9b044f5aac78024b416b0ca, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/2fcceb6a405647d1ac5bbc9529b36df5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/a301c9e4211d45e5a4fb8a9853a443da, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/adb7ad8b4493425ea5300bc1369d558c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/4107b42c00b74571918c8d9c6f0c2da7] to archive 2023-03-27 04:57:55,224 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:57:55,226 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/50bab4d4ff8e49eba9ffa453f9e25522 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/50bab4d4ff8e49eba9ffa453f9e25522 2023-03-27 04:57:55,227 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/76eb603b9a8e4b4791dcf4760e99b78f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/76eb603b9a8e4b4791dcf4760e99b78f 2023-03-27 04:57:55,228 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/f801182b8e4b4ca9b33b4181ea6a91df to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/f801182b8e4b4ca9b33b4181ea6a91df 2023-03-27 04:57:55,229 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/218903ba93c84d238678c09ada4bdf47 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/218903ba93c84d238678c09ada4bdf47 2023-03-27 04:57:55,230 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/f8b64f2bbe704392b8ba5182f11c03ff to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/f8b64f2bbe704392b8ba5182f11c03ff 2023-03-27 04:57:55,231 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/77d099bab9ff42fbaff40ec9ec172aed to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/77d099bab9ff42fbaff40ec9ec172aed 2023-03-27 04:57:55,232 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/11469f5721004b489903e1bc8616fee5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/11469f5721004b489903e1bc8616fee5 2023-03-27 04:57:55,233 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/6414ceb30694420bb1931ea243db30e7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/6414ceb30694420bb1931ea243db30e7 2023-03-27 04:57:55,233 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/999d816008c34fe7a21d4506fdba4f95 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/999d816008c34fe7a21d4506fdba4f95 2023-03-27 04:57:55,234 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/256c32ec99ae46b7be596d3e27581608 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/256c32ec99ae46b7be596d3e27581608 2023-03-27 04:57:55,235 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/dac8264768564a3786e6776ae8510137 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/dac8264768564a3786e6776ae8510137 2023-03-27 04:57:55,236 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/2bd2bbb38ad14ebbb5680bfd1adae94e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/2bd2bbb38ad14ebbb5680bfd1adae94e 2023-03-27 04:57:55,237 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/f8643f7e208b41de9f6a8c83f97ec581 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/f8643f7e208b41de9f6a8c83f97ec581 2023-03-27 04:57:55,238 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/98f6d21eb43a493a9dedd8a005c5bbe1 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/98f6d21eb43a493a9dedd8a005c5bbe1 2023-03-27 04:57:55,238 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/a9f0e07be9b044f5aac78024b416b0ca to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/a9f0e07be9b044f5aac78024b416b0ca 2023-03-27 04:57:55,239 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/2fcceb6a405647d1ac5bbc9529b36df5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/2fcceb6a405647d1ac5bbc9529b36df5 2023-03-27 04:57:55,240 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/a301c9e4211d45e5a4fb8a9853a443da to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/a301c9e4211d45e5a4fb8a9853a443da 2023-03-27 04:57:55,241 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/adb7ad8b4493425ea5300bc1369d558c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/adb7ad8b4493425ea5300bc1369d558c 2023-03-27 04:57:55,242 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/4107b42c00b74571918c8d9c6f0c2da7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/4107b42c00b74571918c8d9c6f0c2da7 2023-03-27 04:57:55,249 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/4a9bfa1a0d4549aab799356053fedff5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/882b9ea524e74d24862538ebeaf79cc8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/3022553b40534628aa5ff28a957ce511, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/f614555e35d94808bd369bdc776f62a1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/faea3cab2a64481592b10dca3fe8de41, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/fa1034e960de47f09430d02879f13085, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/2fcdaf36d5a14acc88836fd3e7f677da, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/0dff9f56ec1e48328093cbe4e53508ca, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/15803a67f0324cfca3aa862d8a244787, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/0effd4bc255a403ca0be8f27e614e032, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/229f1a1679a34637ae3b21135e989419, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/eaba31854ffe48219106afcb428f6bfe, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/7734b3ae3dec4031a37dd01c02ef3867, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/5f555a74cec84e75a4132da85c041bf1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/063594a3e2a14c4f89f6d8c8736353bd, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/739a79c39f2649f69a05db6ad78c73a6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/3497c874be7648138fa73df46253ec52, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/7adc2bcf4adf4270892e7af81b2650fb, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/ad8480e05bbe46f99f7a1502aa1086f1] to archive 2023-03-27 04:57:55,249 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:57:55,251 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/4a9bfa1a0d4549aab799356053fedff5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/4a9bfa1a0d4549aab799356053fedff5 2023-03-27 04:57:55,252 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/882b9ea524e74d24862538ebeaf79cc8 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/882b9ea524e74d24862538ebeaf79cc8 2023-03-27 04:57:55,254 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/3022553b40534628aa5ff28a957ce511 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/3022553b40534628aa5ff28a957ce511 2023-03-27 04:57:55,255 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/f614555e35d94808bd369bdc776f62a1 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/f614555e35d94808bd369bdc776f62a1 2023-03-27 04:57:55,256 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/faea3cab2a64481592b10dca3fe8de41 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/faea3cab2a64481592b10dca3fe8de41 2023-03-27 04:57:55,257 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/fa1034e960de47f09430d02879f13085 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/fa1034e960de47f09430d02879f13085 2023-03-27 04:57:55,257 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/2fcdaf36d5a14acc88836fd3e7f677da to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/2fcdaf36d5a14acc88836fd3e7f677da 2023-03-27 04:57:55,258 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/0dff9f56ec1e48328093cbe4e53508ca to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/0dff9f56ec1e48328093cbe4e53508ca 2023-03-27 04:57:55,259 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/15803a67f0324cfca3aa862d8a244787 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/15803a67f0324cfca3aa862d8a244787 2023-03-27 04:57:55,260 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/0effd4bc255a403ca0be8f27e614e032 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/0effd4bc255a403ca0be8f27e614e032 2023-03-27 04:57:55,260 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/229f1a1679a34637ae3b21135e989419 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/229f1a1679a34637ae3b21135e989419 2023-03-27 04:57:55,261 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/eaba31854ffe48219106afcb428f6bfe to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/eaba31854ffe48219106afcb428f6bfe 2023-03-27 04:57:55,262 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/7734b3ae3dec4031a37dd01c02ef3867 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/7734b3ae3dec4031a37dd01c02ef3867 2023-03-27 04:57:55,262 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/5f555a74cec84e75a4132da85c041bf1 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/5f555a74cec84e75a4132da85c041bf1 2023-03-27 04:57:55,263 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/063594a3e2a14c4f89f6d8c8736353bd to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/063594a3e2a14c4f89f6d8c8736353bd 2023-03-27 04:57:55,264 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/739a79c39f2649f69a05db6ad78c73a6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/739a79c39f2649f69a05db6ad78c73a6 2023-03-27 04:57:55,265 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/3497c874be7648138fa73df46253ec52 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/3497c874be7648138fa73df46253ec52 2023-03-27 04:57:55,265 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/7adc2bcf4adf4270892e7af81b2650fb to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/7adc2bcf4adf4270892e7af81b2650fb 2023-03-27 04:57:55,266 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/ad8480e05bbe46f99f7a1502aa1086f1 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/ad8480e05bbe46f99f7a1502aa1086f1 2023-03-27 04:57:55,268 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/ba80f5ccfad2408b97bec519458ace2c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/3505c96dc9474b0c8d36e0524be9a133, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8c575df33b18447ea1aba69b8c7e287c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8496954652f34711a9e69c67e95accd3, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/2ef90a4bbdc747aaa51a7351d23ab60d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/74d2864b351c434087c9b40e97ddd5c2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/d400e690a98d4776b7805223b5bb40b7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/5bc8def1bca64d89a1afa121b653efb0, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/070690cfb599475cb5dc5f67a374f223, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/86c6ba31da93457c865cec7399c22a28, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/c3dca3502476411c99de162cbf80b433, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/c5733df9c89f41159c4fb4fd411a24e3, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/126eef92d78a4a7a822c9280fdba7620, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8545af0440654e9e9820dc4a73b91d5e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/13698f80b9b44c0289a8bb72dbaca19f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/c59414c5b92f4ba385a84bf38b2002ad, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/7fdf89c884cc48c6a9182b28212cd384, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/63d99e24251f4a1f9ca024bf2522ee66, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/b0020610f9a14e9ea255572d12e0d761] to archive 2023-03-27 04:57:55,269 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:57:55,270 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/ba80f5ccfad2408b97bec519458ace2c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/ba80f5ccfad2408b97bec519458ace2c 2023-03-27 04:57:55,271 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/3505c96dc9474b0c8d36e0524be9a133 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/3505c96dc9474b0c8d36e0524be9a133 2023-03-27 04:57:55,271 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8c575df33b18447ea1aba69b8c7e287c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8c575df33b18447ea1aba69b8c7e287c 2023-03-27 04:57:55,272 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8496954652f34711a9e69c67e95accd3 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8496954652f34711a9e69c67e95accd3 2023-03-27 04:57:55,273 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/2ef90a4bbdc747aaa51a7351d23ab60d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/2ef90a4bbdc747aaa51a7351d23ab60d 2023-03-27 04:57:55,273 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/74d2864b351c434087c9b40e97ddd5c2 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/74d2864b351c434087c9b40e97ddd5c2 2023-03-27 04:57:55,274 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/d400e690a98d4776b7805223b5bb40b7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/d400e690a98d4776b7805223b5bb40b7 2023-03-27 04:57:55,275 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/5bc8def1bca64d89a1afa121b653efb0 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/5bc8def1bca64d89a1afa121b653efb0 2023-03-27 04:57:55,275 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/070690cfb599475cb5dc5f67a374f223 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/070690cfb599475cb5dc5f67a374f223 2023-03-27 04:57:55,276 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/86c6ba31da93457c865cec7399c22a28 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/86c6ba31da93457c865cec7399c22a28 2023-03-27 04:57:55,276 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/c3dca3502476411c99de162cbf80b433 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/c3dca3502476411c99de162cbf80b433 2023-03-27 04:57:55,277 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/c5733df9c89f41159c4fb4fd411a24e3 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/c5733df9c89f41159c4fb4fd411a24e3 2023-03-27 04:57:55,278 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/126eef92d78a4a7a822c9280fdba7620 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/126eef92d78a4a7a822c9280fdba7620 2023-03-27 04:57:55,279 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8545af0440654e9e9820dc4a73b91d5e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8545af0440654e9e9820dc4a73b91d5e 2023-03-27 04:57:55,279 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/13698f80b9b44c0289a8bb72dbaca19f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/13698f80b9b44c0289a8bb72dbaca19f 2023-03-27 04:57:55,280 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/c59414c5b92f4ba385a84bf38b2002ad to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/c59414c5b92f4ba385a84bf38b2002ad 2023-03-27 04:57:55,280 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/7fdf89c884cc48c6a9182b28212cd384 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/7fdf89c884cc48c6a9182b28212cd384 2023-03-27 04:57:55,281 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/63d99e24251f4a1f9ca024bf2522ee66 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/63d99e24251f4a1f9ca024bf2522ee66 2023-03-27 04:57:55,282 DEBUG [StoreCloser-TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/b0020610f9a14e9ea255572d12e0d761 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/b0020610f9a14e9ea255572d12e0d761 2023-03-27 04:57:55,285 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/recovered.edits/937.seqid, newMaxSeqId=937, maxSeqId=4 2023-03-27 04:57:55,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc. 2023-03-27 04:57:55,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 682db1751885437677f00c4ab8bf8ffc: 2023-03-27 04:57:55,288 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:55,288 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=682db1751885437677f00c4ab8bf8ffc, regionState=CLOSED 2023-03-27 04:57:55,288 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893075288"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679893075288"}]},"ts":"1679893075288"} 2023-03-27 04:57:55,290 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-27 04:57:55,290 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 682db1751885437677f00c4ab8bf8ffc, server=jenkins-hbase20.apache.org,44923,1679892955505 in 3.3940 sec 2023-03-27 04:57:55,291 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-27 04:57:55,291 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=682db1751885437677f00c4ab8bf8ffc, UNASSIGN in 3.3970 sec 2023-03-27 04:57:55,292 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893075292"}]},"ts":"1679893075292"} 2023-03-27 04:57:55,293 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-27 04:57:55,294 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-27 04:57:55,295 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.4180 sec 2023-03-27 04:57:55,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-27 04:57:55,992 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-27 04:57:55,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-27 04:57:55,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:55,996 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:55,996 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:55,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-27 04:57:56,002 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:56,005 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/recovered.edits] 2023-03-27 04:57:56,009 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/1ab57e58917e4307a53d3662820bf5b4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/1ab57e58917e4307a53d3662820bf5b4 2023-03-27 04:57:56,011 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/7c742b9df205427e839215be98a80b2e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/7c742b9df205427e839215be98a80b2e 2023-03-27 04:57:56,012 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/ed7e000b27064395ae023482de51f5ff to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/A/ed7e000b27064395ae023482de51f5ff 2023-03-27 04:57:56,016 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/88183f65fa5c44ed80ba63e9dc95b22d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/88183f65fa5c44ed80ba63e9dc95b22d 2023-03-27 04:57:56,017 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/ee33f77cf0a544189cd40c5342088e2b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/ee33f77cf0a544189cd40c5342088e2b 2023-03-27 04:57:56,018 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/f8a71c81730c4cec815335991606f2b2 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/B/f8a71c81730c4cec815335991606f2b2 2023-03-27 04:57:56,019 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/082b1b6c56ed44a6b3e055b26cb65fc9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/082b1b6c56ed44a6b3e055b26cb65fc9 2023-03-27 04:57:56,020 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8cf50da2f701431fb180e7e83379e982 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/8cf50da2f701431fb180e7e83379e982 2023-03-27 04:57:56,021 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/af8c40fd811b4d0290b5bbaec93bba30 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/C/af8c40fd811b4d0290b5bbaec93bba30 2023-03-27 04:57:56,023 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/recovered.edits/937.seqid to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc/recovered.edits/937.seqid 2023-03-27 04:57:56,024 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/682db1751885437677f00c4ab8bf8ffc 2023-03-27 04:57:56,024 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-27 04:57:56,024 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-27 04:57:56,025 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-27 04:57:56,026 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032713328d0b8f034ad096d42ce7988b062f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032713328d0b8f034ad096d42ce7988b062f 2023-03-27 04:57:56,027 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032716bf1603d7ff43428756b14a4e8fae40 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032716bf1603d7ff43428756b14a4e8fae40 2023-03-27 04:57:56,028 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303271b9df826dc3e476db13274ae207edcff to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303271b9df826dc3e476db13274ae207edcff 2023-03-27 04:57:56,029 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303271dd8b8f6840b44329473c876e03056bb to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303271dd8b8f6840b44329473c876e03056bb 2023-03-27 04:57:56,030 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032732d007d04a7d4d7396314a06d123c22d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032732d007d04a7d4d7396314a06d123c22d 2023-03-27 04:57:56,031 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303273d2d1827a84e4c0c9d87b20a64dd974a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303273d2d1827a84e4c0c9d87b20a64dd974a 2023-03-27 04:57:56,032 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327471fd2dc5708464e939d3a3453024faa to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327471fd2dc5708464e939d3a3453024faa 2023-03-27 04:57:56,032 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032765fc3a4a9281465bb4f102f3fc2ebe6d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032765fc3a4a9281465bb4f102f3fc2ebe6d 2023-03-27 04:57:56,033 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303276c3d11b4b5a34996b8eca1b417d52f9d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303276c3d11b4b5a34996b8eca1b417d52f9d 2023-03-27 04:57:56,034 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303277780152d4beb4b5b9842359b59beeac6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303277780152d4beb4b5b9842359b59beeac6 2023-03-27 04:57:56,035 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327a82405624931403d895047dee00246dd to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327a82405624931403d895047dee00246dd 2023-03-27 04:57:56,036 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327ae3868807f6c453eab15083d6c32d9f7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327ae3868807f6c453eab15083d6c32d9f7 2023-03-27 04:57:56,036 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327be0ca5c201c94fb695f37451aaaaf7f6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327be0ca5c201c94fb695f37451aaaaf7f6 2023-03-27 04:57:56,037 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327c427972338bb41749226ab35ad222d1c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327c427972338bb41749226ab35ad222d1c 2023-03-27 04:57:56,038 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327d0681f7078ff4fdaa1734a2b3615463a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327d0681f7078ff4fdaa1734a2b3615463a 2023-03-27 04:57:56,039 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e2f63be8409841a883e88a6aceb01a51 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e2f63be8409841a883e88a6aceb01a51 2023-03-27 04:57:56,039 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e5eaacb2a470487a91354c15ec0f5d9e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e5eaacb2a470487a91354c15ec0f5d9e 2023-03-27 04:57:56,040 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327f2f5c6b7888d420ea6949822542365f5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327f2f5c6b7888d420ea6949822542365f5 2023-03-27 04:57:56,040 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-27 04:57:56,042 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:56,043 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-27 04:57:56,044 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-27 04:57:56,045 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:56,045 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-27 04:57:56,045 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679893076045"}]},"ts":"9223372036854775807"} 2023-03-27 04:57:56,047 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-27 04:57:56,047 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 682db1751885437677f00c4ab8bf8ffc, NAME => 'TestAcidGuarantees,,1679893048036.682db1751885437677f00c4ab8bf8ffc.', STARTKEY => '', ENDKEY => ''}] 2023-03-27 04:57:56,047 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-27 04:57:56,047 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679893076047"}]},"ts":"9223372036854775807"} 2023-03-27 04:57:56,048 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-27 04:57:56,049 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:56,050 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 56 msec 2023-03-27 04:57:56,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-27 04:57:56,098 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-27 04:57:56,112 INFO [Listener at localhost.localdomain/43311] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 228), OpenFileDescriptor=531 (was 559), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=412 (was 463), ProcessCount=167 (was 167), AvailableMemoryMB=879 (was 849) - AvailableMemoryMB LEAK? - 2023-03-27 04:57:56,125 INFO [Listener at localhost.localdomain/43311] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=531, MaxFileDescriptor=60000, SystemLoadAverage=412, ProcessCount=167, AvailableMemoryMB=878 2023-03-27 04:57:56,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-27 04:57:56,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-27 04:57:56,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-27 04:57:56,130 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-27 04:57:56,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-27 04:57:56,130 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-27 04:57:56,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-27 04:57:56,132 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:57:56,132 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f empty. 2023-03-27 04:57:56,132 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:57:56,132 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-27 04:57:56,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-27 04:57:56,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-27 04:57:56,541 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-27 04:57:56,544 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 72e4f7cba88d8e5eeeccc51fc07b799f, NAME => 'TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp 2023-03-27 04:57:56,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-27 04:57:56,961 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:57:56,961 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 72e4f7cba88d8e5eeeccc51fc07b799f, disabling compactions & flushes 2023-03-27 04:57:56,961 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:56,961 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:56,961 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. after waiting 0 ms 2023-03-27 04:57:56,961 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:56,961 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:56,961 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:57:56,964 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-27 04:57:56,965 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893076965"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679893076965"}]},"ts":"1679893076965"} 2023-03-27 04:57:56,966 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-27 04:57:56,967 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-27 04:57:56,967 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893076967"}]},"ts":"1679893076967"} 2023-03-27 04:57:56,968 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-27 04:57:56,972 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=72e4f7cba88d8e5eeeccc51fc07b799f, ASSIGN}] 2023-03-27 04:57:56,974 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=72e4f7cba88d8e5eeeccc51fc07b799f, ASSIGN 2023-03-27 04:57:56,975 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=72e4f7cba88d8e5eeeccc51fc07b799f, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,44923,1679892955505; forceNewPlan=false, retain=false 2023-03-27 04:57:57,127 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=72e4f7cba88d8e5eeeccc51fc07b799f, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:57,127 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893077127"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679893077127"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679893077127"}]},"ts":"1679893077127"} 2023-03-27 04:57:57,130 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:57:57,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-27 04:57:57,292 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:57,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 72e4f7cba88d8e5eeeccc51fc07b799f, NAME => 'TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.', STARTKEY => '', ENDKEY => ''} 2023-03-27 04:57:57,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:57:57,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:57:57,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:57:57,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:57:57,295 INFO [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:57:57,297 DEBUG [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A 2023-03-27 04:57:57,297 DEBUG [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A 2023-03-27 04:57:57,297 INFO [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:57:57,297 INFO [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 72e4f7cba88d8e5eeeccc51fc07b799f columnFamilyName A 2023-03-27 04:57:57,298 INFO [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] regionserver.HStore(310): Store=72e4f7cba88d8e5eeeccc51fc07b799f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:57:57,298 INFO [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:57:57,300 DEBUG [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B 2023-03-27 04:57:57,300 DEBUG [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B 2023-03-27 04:57:57,301 INFO [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:57:57,301 INFO [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 72e4f7cba88d8e5eeeccc51fc07b799f columnFamilyName B 2023-03-27 04:57:57,302 INFO [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] regionserver.HStore(310): Store=72e4f7cba88d8e5eeeccc51fc07b799f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:57:57,302 INFO [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:57:57,304 DEBUG [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C 2023-03-27 04:57:57,304 DEBUG [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C 2023-03-27 04:57:57,304 INFO [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:57:57,304 INFO [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 72e4f7cba88d8e5eeeccc51fc07b799f columnFamilyName C 2023-03-27 04:57:57,305 INFO [StoreOpener-72e4f7cba88d8e5eeeccc51fc07b799f-1] regionserver.HStore(310): Store=72e4f7cba88d8e5eeeccc51fc07b799f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:57:57,305 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:57,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:57:57,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:57:57,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-27 04:57:57,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:57:57,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-27 04:57:57,313 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 72e4f7cba88d8e5eeeccc51fc07b799f; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71555590, jitterRate=0.06626138091087341}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-27 04:57:57,313 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:57:57,313 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., pid=49, masterSystemTime=1679893077284 2023-03-27 04:57:57,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:57,314 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:57,314 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=72e4f7cba88d8e5eeeccc51fc07b799f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:57,314 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893077314"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679893077314"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679893077314"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679893077314"}]},"ts":"1679893077314"} 2023-03-27 04:57:57,316 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-27 04:57:57,316 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 in 185 msec 2023-03-27 04:57:57,317 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-27 04:57:57,317 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=72e4f7cba88d8e5eeeccc51fc07b799f, ASSIGN in 344 msec 2023-03-27 04:57:57,317 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-27 04:57:57,318 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893077317"}]},"ts":"1679893077317"} 2023-03-27 04:57:57,318 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-27 04:57:57,319 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-27 04:57:57,320 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1930 sec 2023-03-27 04:57:58,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-27 04:57:58,243 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-27 04:57:58,244 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x09e26d67 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:58,252 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55409eb8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:58,253 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:58,256 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43068, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:58,257 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-27 04:57:58,259 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59020, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-27 04:57:58,261 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x1a885842 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:58,268 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9b9721, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:58,268 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x02e8c6f6 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:58,276 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a7ac4a1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:58,277 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x2cb7a788 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:58,299 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@682c0b9e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:58,299 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x41ccedcb to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:58,306 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50d6899d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:58,307 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x40e6f624 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:58,318 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6662e946, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:58,318 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x5d7deb76 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:58,336 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c240146, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:58,337 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x0f4b7807 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:58,344 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d5f04c3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:58,344 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x31fc0e3f to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:58,351 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6523d347, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:58,352 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x79db57d4 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:58,359 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28072326, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:58,360 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x0e56ea94 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:57:58,368 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d90797d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:57:58,375 DEBUG [hconnection-0x63e8603b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:58,376 DEBUG [hconnection-0x58539e68-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:58,376 DEBUG [hconnection-0x5002ab85-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:58,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:58,376 DEBUG [hconnection-0x6040b8ee-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:58,376 DEBUG [hconnection-0x604da57d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:58,377 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43080, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:58,381 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43102, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:58,381 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43130, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:58,381 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43118, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:58,381 DEBUG [hconnection-0x30d17a94-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:58,381 DEBUG [hconnection-0xa03d9b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:58,381 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43082, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:58,381 DEBUG [hconnection-0x47bf03e0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:58,383 DEBUG [hconnection-0x1234dff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:58,383 DEBUG [hconnection-0x1a50bd28-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:57:58,383 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43142, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:58,387 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43138, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:58,387 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43140, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:58,387 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43148, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:58,387 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43150, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:57:58,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:58,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:58,389 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:58,389 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:58,389 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:58,389 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:58,389 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,389 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,390 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,390 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:58,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,392 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,392 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:58,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:58,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:58,398 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-27 04:57:58,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:57:58,401 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:58,401 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:58,401 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:58,402 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:58,402 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:58,402 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:58,402 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:57:58,402 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:57:58,402 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:58,402 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:57:58,402 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:58,402 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:58,402 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:58,402 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,402 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:57:58,402 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:58,402 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:57:58,402 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:58,405 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:58,405 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:58,410 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/a53484e06a5046c389e6f0a46293d015 2023-03-27 04:57:58,411 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,411 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:58,411 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,411 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:58,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:58,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ab336ac[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:58,413 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:58,413 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,413 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,413 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,413 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:58,413 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:58,414 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:58,414 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:58,414 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,415 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:58,415 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:58,415 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:58,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893138410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893138414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:58,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:58,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:58,417 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:58,417 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:58,417 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:58,417 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:58,417 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:58,418 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,418 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,418 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:58,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:58,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:58,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893138415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893138416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893138416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,419 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,420 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:58,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,421 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,421 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:58,421 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,421 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:58,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:58,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:58,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:58,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:58,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:58,421 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:58,421 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,422 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,422 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:58,422 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:58,422 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:58,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:58,422 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,422 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,426 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/7d320b4903ed49b79c751e1cb7a5a995 2023-03-27 04:57:58,444 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/168feefd03634291b02e629e9c8404d3 2023-03-27 04:57:58,446 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/a53484e06a5046c389e6f0a46293d015 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a53484e06a5046c389e6f0a46293d015 2023-03-27 04:57:58,449 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a53484e06a5046c389e6f0a46293d015, entries=150, sequenceid=18, filesize=11.6 K 2023-03-27 04:57:58,449 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/7d320b4903ed49b79c751e1cb7a5a995 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/7d320b4903ed49b79c751e1cb7a5a995 2023-03-27 04:57:58,452 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/7d320b4903ed49b79c751e1cb7a5a995, entries=150, sequenceid=18, filesize=11.6 K 2023-03-27 04:57:58,455 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/168feefd03634291b02e629e9c8404d3 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/168feefd03634291b02e629e9c8404d3 2023-03-27 04:57:58,459 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/168feefd03634291b02e629e9c8404d3, entries=150, sequenceid=18, filesize=11.6 K 2023-03-27 04:57:58,459 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 72e4f7cba88d8e5eeeccc51fc07b799f in 61ms, sequenceid=18, compaction requested=false 2023-03-27 04:57:58,459 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-27 04:57:58,460 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:57:58,522 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:58,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:58,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:57:58,523 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-27 04:57:58,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:58,524 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:57:58,524 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:58,524 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:57:58,524 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:58,525 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:57:58,525 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:58,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:58,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:58,525 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:58,525 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:58,525 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:58,525 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:58,526 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,526 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,527 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:58,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:58,528 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,528 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,528 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:58,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:58,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:58,529 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:58,529 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:58,529 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:58,529 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:58,529 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:58,529 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:58,530 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:57:58,530 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:57:58,530 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:58,530 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:58,530 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:58,530 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:58,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:58,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,538 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/a8c23eb358f84989a413ecd6143db071 2023-03-27 04:57:58,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893138535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893138535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893138536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,542 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,542 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:58,543 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:58,543 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:58,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:58,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a8d43d1[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:58,544 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:58,544 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,545 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893138538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,544 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893138538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,546 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:58,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,547 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:58,547 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:58,547 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:58,547 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,547 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:58,547 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:58,548 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:58,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:58,549 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:58,549 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:58,549 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:58,549 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:58,549 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:58,550 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,550 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:58,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,550 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:58,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:58,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:58,551 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,552 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,554 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,554 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,554 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:58,554 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:58,554 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:58,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:58,554 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:58,554 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:58,554 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,554 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:58,555 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,554 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:58,555 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:58,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:58,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893138639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893138641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893138641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893138647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893138647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,655 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:58,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:58,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:58,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:58,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:58,657 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:58,657 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:58,657 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:58,657 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:58,658 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,658 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,658 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:58,658 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:58,659 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,659 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:58,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:58,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:58,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:58,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:58,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:58,660 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:58,660 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:58,660 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:58,660 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:57:58,660 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:57:58,660 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:58,660 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:58,660 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:58,660 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,660 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:58,660 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:58,661 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:58,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:58,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:58,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@651c8a99[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:58,662 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:58,662 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,663 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,663 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:58,663 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:58,663 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:58,663 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,663 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:58,663 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,664 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:58,664 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:58,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:58,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:58,664 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:58,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:58,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:58,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:58,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:58,666 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:58,666 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:58,666 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:58,666 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:58,666 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:58,668 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,668 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:58,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,668 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:58,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:58,669 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,669 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,671 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,671 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,671 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,671 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,671 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,671 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,671 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,671 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:58,671 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:58,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:58,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:58,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:58,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:58,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:58,671 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:58,671 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:58,671 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:58,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:58,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:58,772 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:58,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:58,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:58,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:58,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:58,774 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:58,774 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:58,774 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:58,774 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:58,774 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,774 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,775 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:58,775 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:58,775 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,775 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:58,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:58,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:58,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:58,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:58,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:58,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:58,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:58,776 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:58,776 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:57:58,776 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:57:58,776 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:58,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:58,777 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:58,777 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,777 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:58,777 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:58,777 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,777 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:58,777 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,778 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:58,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:58,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f8dd454[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:58,779 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:58,779 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,779 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,779 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:58,779 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:58,779 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:58,780 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,780 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:58,780 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:58,780 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,780 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:58,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:58,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:58,780 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:58,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:58,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:58,782 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:58,782 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:58,782 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:58,782 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:58,782 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:58,783 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,783 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:58,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,783 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:58,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:58,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:58,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:58,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:58,783 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:58,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,784 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:58,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:58,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:58,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:58,786 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:58,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:58,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:58,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:58,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,786 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:58,786 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:58,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:58,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:58,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893138842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893138843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893138844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893138850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:58,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893138850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,886 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:58,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:58,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:58,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:58,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:58,889 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:58,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:58,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:58,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:58,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,890 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,890 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:58,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:58,890 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,890 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:58,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:58,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:58,891 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:58,891 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:58,891 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:58,891 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:58,891 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:58,891 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:58,891 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:57:58,891 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:57:58,891 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:58,891 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:58,891 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:58,891 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,891 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:58,891 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:58,892 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:58,892 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:58,892 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,892 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:58,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:58,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cd8815f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:58,893 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:58,893 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,894 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,894 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:58,894 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:58,894 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,894 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:58,894 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:58,894 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,894 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:58,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:58,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:58,894 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:58,895 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:58,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:58,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:58,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:58,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:58,896 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:58,896 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:58,896 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:58,896 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:58,896 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:58,897 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:58,897 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,897 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,897 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:58,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,898 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:58,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:58,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:58,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:58,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:58,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:58,900 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:58,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:58,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:58,900 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:58,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,900 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:58,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:58,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:58,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:58,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:58,946 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/e7889c114fd3434692dc5371c61ea742 2023-03-27 04:57:59,000 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:59,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:59,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:59,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:59,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:59,005 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:59,005 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:59,005 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:59,005 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:59,005 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,005 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,006 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,006 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:59,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:59,006 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,006 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:59,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:59,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:59,007 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:59,007 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:59,007 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:59,007 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:59,007 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:59,007 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:59,007 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:57:59,008 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:57:59,008 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:59,008 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:59,008 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:59,008 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,008 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:59,008 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:59,008 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,008 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:59,009 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,009 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:59,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:59,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ccbd6de[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:59,010 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:59,010 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,010 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,010 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:59,010 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:59,010 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:59,010 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,011 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:59,011 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:59,011 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:59,011 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,011 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:59,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:59,013 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:59,013 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:59,013 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:59,013 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:59,013 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:59,014 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,014 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:59,014 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:59,014 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,015 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:59,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:59,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:59,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:59,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:59,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:59,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,018 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,018 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,018 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,018 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,018 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,018 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:59,018 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:59,018 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:59,019 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:59,119 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:59,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:59,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:59,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:59,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:59,122 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:59,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:59,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:59,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:59,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,123 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:59,123 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:59,123 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,123 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:59,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:59,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:59,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:59,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:59,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:59,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:59,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:59,127 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:59,127 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:57:59,127 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:57:59,127 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:59,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:59,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:59,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:59,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:59,128 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,128 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:59,128 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,128 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:59,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:59,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d82c39c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:59,129 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:59,129 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,130 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,130 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,130 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,130 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:59,130 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:59,130 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:59,131 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,131 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:59,131 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:59,131 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:59,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:59,133 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:59,133 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:59,133 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:59,133 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:59,133 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:59,134 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,134 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:59,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,134 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:59,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,136 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,137 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,138 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:59,138 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:59,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:59,138 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:59,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:59,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:59,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:59,138 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:59,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:59,138 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:59,139 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:59,139 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,139 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,139 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,139 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,139 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,139 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:59,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893139151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893139151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893139151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893139155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893139157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,239 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:59,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:59,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:59,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:59,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:59,242 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:59,242 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:59,242 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:59,242 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:59,242 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,242 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,243 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,243 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:59,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:59,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:59,243 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,243 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:59,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:59,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:59,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:59,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:59,244 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:59,244 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:59,244 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:59,244 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:59,244 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:59,244 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:59,244 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:57:59,244 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:57:59,244 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:59,244 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:59,244 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:59,244 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,244 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:59,244 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:59,245 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,245 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:59,245 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,246 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:59,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:59,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cec870c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:59,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:59,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,247 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,247 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:59,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:59,247 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:59,247 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:59,248 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:59,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:59,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:59,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:59,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:59,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:59,250 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:59,250 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:59,250 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:59,250 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:59,250 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:59,251 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,251 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:59,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,251 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:59,251 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,252 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:59,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:59,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:59,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:59,254 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:59,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:59,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:59,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:59,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:59,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:59,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,254 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:59,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,254 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:59,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,355 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:59,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:59,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:59,356 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/8927d160474246b7ad1c460ab26eab58 2023-03-27 04:57:59,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:59,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:59,357 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:59,357 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:59,357 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:59,357 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:59,358 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,358 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:59,358 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:59,359 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,359 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:59,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:59,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:59,359 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:59,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:59,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:59,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:59,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:59,360 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:59,360 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:57:59,360 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:57:59,360 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/a8c23eb358f84989a413ecd6143db071 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a8c23eb358f84989a413ecd6143db071 2023-03-27 04:57:59,360 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:59,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:57:59,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:57:59,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:57:59,360 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:57:59,361 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:59,361 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:59,361 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,362 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:57:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:59,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:57:59,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4863bb7[Count = 0] remaining members to acquire global barrier 2023-03-27 04:57:59,362 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:57:59,363 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,363 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a8c23eb358f84989a413ecd6143db071, entries=250, sequenceid=38, filesize=16.2 K 2023-03-27 04:57:59,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/e7889c114fd3434692dc5371c61ea742 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/e7889c114fd3434692dc5371c61ea742 2023-03-27 04:57:59,364 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:57:59,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:57:59,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:57:59,364 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,364 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:57:59,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:57:59,364 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:59,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:57:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:59,366 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/e7889c114fd3434692dc5371c61ea742, entries=150, sequenceid=38, filesize=11.6 K 2023-03-27 04:57:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:57:59,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:57:59,367 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:57:59,367 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:57:59,367 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:57:59,367 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:57:59,367 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/8927d160474246b7ad1c460ab26eab58 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/8927d160474246b7ad1c460ab26eab58 2023-03-27 04:57:59,367 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:57:59,368 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,368 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:57:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:57:59,368 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:59,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:57:59,369 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:57:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,370 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:57:59,370 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/8927d160474246b7ad1c460ab26eab58, entries=150, sequenceid=38, filesize=11.6 K 2023-03-27 04:57:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:57:59,371 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=73.80 KB/75570 for 72e4f7cba88d8e5eeeccc51fc07b799f in 848ms, sequenceid=38, compaction requested=false 2023-03-27 04:57:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,371 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:57:59,371 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:59,371 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:57:59,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:59,371 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:57:59,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:57:59,371 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:57:59,371 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,372 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:59,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:57:59,372 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,372 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:57:59,372 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:57:59,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:57:59,372 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:57:59,372 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:57:59,372 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,472 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:57:59,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:57:59,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:57:59,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:57:59,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:57:59,474 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:57:59,474 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:59,474 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:57:59,474 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:57:59,475 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,475 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,475 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:57:59,475 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:57:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:57:59,476 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,476 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:57:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:57:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:57:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:57:59,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:57:59,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:57:59,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:57:59,477 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:57:59,477 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:57:59,477 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:57:59,477 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:57:59,477 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:57:59,477 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:57:59,477 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:57:59,477 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-27 04:57:59,477 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:57:59,477 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:59,477 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:57:59,477 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:59,477 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:57:59,477 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:57:59,484 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/70349533f17a4e21a6a6cfbd95336950 2023-03-27 04:57:59,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:57:59,657 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:57:59,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893139669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893139670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893139675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893139675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893139676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893139776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893139777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893139781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893139785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893139785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,894 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/082fc3a8016f44ad8708adaaf1b3cc02 2023-03-27 04:57:59,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893139980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893139980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893139985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893139989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:57:59,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:57:59,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893139990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:00,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893140286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:00,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893140287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,301 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/c8aad07a200847d592082c699cd4c101 2023-03-27 04:58:00,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:00,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893140293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,304 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/70349533f17a4e21a6a6cfbd95336950 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/70349533f17a4e21a6a6cfbd95336950 2023-03-27 04:58:00,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:00,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893140297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:00,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893140297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,307 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/70349533f17a4e21a6a6cfbd95336950, entries=150, sequenceid=52, filesize=11.6 K 2023-03-27 04:58:00,308 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/082fc3a8016f44ad8708adaaf1b3cc02 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/082fc3a8016f44ad8708adaaf1b3cc02 2023-03-27 04:58:00,310 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/082fc3a8016f44ad8708adaaf1b3cc02, entries=150, sequenceid=52, filesize=11.6 K 2023-03-27 04:58:00,311 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/c8aad07a200847d592082c699cd4c101 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/c8aad07a200847d592082c699cd4c101 2023-03-27 04:58:00,313 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/c8aad07a200847d592082c699cd4c101, entries=150, sequenceid=52, filesize=11.6 K 2023-03-27 04:58:00,314 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 72e4f7cba88d8e5eeeccc51fc07b799f in 837ms, sequenceid=52, compaction requested=true 2023-03-27 04:58:00,314 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:00,314 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:00,314 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:00,314 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:00,314 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,314 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:00,314 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:00,315 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,315 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:00,315 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:00,315 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:00,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:00,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@265960df[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:00,316 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:00,316 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:00,317 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:00,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:00,317 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:00,317 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:00,317 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:00,317 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,317 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:00,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:00,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:00,318 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:00,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:00,320 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:00,320 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:00,320 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:00,320 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:00,320 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:00,322 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:00,322 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:00,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:00,322 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:00,322 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:00,323 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,324 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,324 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:00,324 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:00,324 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,324 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:00,324 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:00,324 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:00,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:00,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:00,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:00,324 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:00,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:00,324 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:00,325 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:00,325 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:00,325 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:00,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:00,425 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:00,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:00,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:00,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:00,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:00,427 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:00,427 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:00,427 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:00,427 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:00,428 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:00,428 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:00,428 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:00,428 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:00,429 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:00,429 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:00,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:00,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:00,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:00,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:00,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:00,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:00,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:00,430 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:00,430 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:00,430 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:00,430 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:00,430 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-27 04:58:00,430 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:00,430 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:00,430 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:00,430 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:00,430 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:00,430 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:00,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:00,801 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:00,810 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-27 04:58:00,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:00,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893140816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:00,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893140817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:00,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893140818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:00,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893140819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:00,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893140819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,836 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/c3f800bdb4cd49688498e7b66aa8df00 2023-03-27 04:58:00,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:00,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893140924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:00,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893140924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:00,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893140924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:00,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893140924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:00,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:00,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893140925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:01,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893141128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:01,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893141128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:01,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893141129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:01,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:01,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893141129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893141131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,248 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/815406931936497ea1f481d4389046f5 2023-03-27 04:58:01,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:01,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893141436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:01,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893141438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:01,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893141438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:01,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893141439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:01,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893141440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,656 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/45bf3280e00b4561ae8bfe123b5fedda 2023-03-27 04:58:01,659 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/c3f800bdb4cd49688498e7b66aa8df00 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/c3f800bdb4cd49688498e7b66aa8df00 2023-03-27 04:58:01,662 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/c3f800bdb4cd49688498e7b66aa8df00, entries=150, sequenceid=70, filesize=11.6 K 2023-03-27 04:58:01,662 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/815406931936497ea1f481d4389046f5 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/815406931936497ea1f481d4389046f5 2023-03-27 04:58:01,665 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/815406931936497ea1f481d4389046f5, entries=150, sequenceid=70, filesize=11.6 K 2023-03-27 04:58:01,665 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/45bf3280e00b4561ae8bfe123b5fedda as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/45bf3280e00b4561ae8bfe123b5fedda 2023-03-27 04:58:01,668 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/45bf3280e00b4561ae8bfe123b5fedda, entries=150, sequenceid=70, filesize=11.6 K 2023-03-27 04:58:01,668 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 72e4f7cba88d8e5eeeccc51fc07b799f in 1238ms, sequenceid=70, compaction requested=true 2023-03-27 04:58:01,669 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:01,669 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:01,669 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:01,669 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:01,669 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,669 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:01,669 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:01,670 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,670 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:01,670 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:01,670 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:01,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:01,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d2cc9ab[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:01,671 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:01,671 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:01,671 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:01,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:01,671 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:01,671 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:01,672 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:01,672 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,672 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:01,672 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:01,672 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:01,672 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:01,672 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:01,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:01,675 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:01,676 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:01,676 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:01,676 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:01,676 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:01,677 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:01,677 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:01,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:01,677 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:01,677 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:01,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:01,678 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,680 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:01,680 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:01,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:01,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:01,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:01,680 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:01,680 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:01,680 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:01,680 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:01,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:01,680 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:01,680 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:01,680 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,680 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:01,680 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:01,680 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:01,780 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:01,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:01,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:01,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:01,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:01,782 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:01,783 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:01,783 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:01,783 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:01,783 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:01,783 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:01,784 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:01,784 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:01,784 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,784 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:01,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:01,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:01,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:01,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:01,785 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:01,785 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:01,785 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:01,785 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:01,785 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:01,785 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:01,785 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:01,785 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-27 04:58:01,785 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:01,785 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:01,785 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:01,785 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:01,785 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:01,786 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:01,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:01,945 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:01,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:01,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893141961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:01,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893141967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:01,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893141968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:01,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893141971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:01,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:01,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893141972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:02,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893142073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:02,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893142074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:02,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893142076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:02,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893142080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:02,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893142080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,192 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/ad12db80f0f948218ecf25bfee5ad2f0 2023-03-27 04:58:02,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:02,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893142277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:02,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893142278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:02,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893142278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:02,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893142284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:02,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893142285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:02,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893142580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:02,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893142581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:02,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893142583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:02,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893142589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:02,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893142589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,600 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/c58379f2d71a4d6793318f94b8ec528f 2023-03-27 04:58:02,608 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/4f8894ce90034c14a1a7bb88dd8ff974 2023-03-27 04:58:02,611 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/ad12db80f0f948218ecf25bfee5ad2f0 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/ad12db80f0f948218ecf25bfee5ad2f0 2023-03-27 04:58:02,613 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/ad12db80f0f948218ecf25bfee5ad2f0, entries=150, sequenceid=84, filesize=11.6 K 2023-03-27 04:58:02,614 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/c58379f2d71a4d6793318f94b8ec528f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/c58379f2d71a4d6793318f94b8ec528f 2023-03-27 04:58:02,616 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/c58379f2d71a4d6793318f94b8ec528f, entries=150, sequenceid=84, filesize=11.6 K 2023-03-27 04:58:02,617 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/4f8894ce90034c14a1a7bb88dd8ff974 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/4f8894ce90034c14a1a7bb88dd8ff974 2023-03-27 04:58:02,619 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/4f8894ce90034c14a1a7bb88dd8ff974, entries=150, sequenceid=84, filesize=11.6 K 2023-03-27 04:58:02,620 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 72e4f7cba88d8e5eeeccc51fc07b799f in 835ms, sequenceid=84, compaction requested=true 2023-03-27 04:58:02,620 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:02,620 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:02,620 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:02,620 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:02,620 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,620 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:02,620 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:02,621 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:02,621 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:02,621 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:02,621 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:02,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:02,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:02,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1520f3e5[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:02,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:02,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:02,623 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:02,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:02,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:02,623 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:02,623 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:02,623 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:02,623 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,623 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:02,624 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,624 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:02,624 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:02,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:02,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:02,624 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:02,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:02,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:02,626 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:02,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:02,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:02,626 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:02,626 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:02,627 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:02,627 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:02,627 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:02,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:02,627 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,628 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:02,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:02,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:02,629 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:02,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:02,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:02,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:02,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:02,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:02,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:02,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:02,630 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:02,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:02,630 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:02,630 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:02,630 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:02,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:02,630 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:02,730 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:02,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:02,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:02,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:02,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:02,734 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:02,734 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:02,734 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:02,734 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:02,734 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:02,734 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:02,735 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,735 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:02,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:02,735 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:02,735 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:02,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:02,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:02,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:02,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:02,736 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:02,736 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:02,736 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:02,736 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:02,736 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:02,736 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:02,736 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:02,736 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-27 04:58:02,736 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:02,736 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:02,736 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:02,736 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:02,736 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:02,736 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:03,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:03,092 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:03,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893143101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893143102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893143103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893143104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893143111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,143 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/5d4ef7d568d14467ab640dd05d9fc481 2023-03-27 04:58:03,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893143211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893143211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893143215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893143215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893143219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893143421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893143421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893143421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893143421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893143428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,554 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/54e2e6fa1f664f03a9cf945be7c2cf88 2023-03-27 04:58:03,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893143726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893143727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893143728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893143729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:03,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893143737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,961 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/1f2e632d223f4ecdbcc2d7bd40012dad 2023-03-27 04:58:03,964 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/5d4ef7d568d14467ab640dd05d9fc481 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/5d4ef7d568d14467ab640dd05d9fc481 2023-03-27 04:58:03,967 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/5d4ef7d568d14467ab640dd05d9fc481, entries=150, sequenceid=102, filesize=11.6 K 2023-03-27 04:58:03,967 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/54e2e6fa1f664f03a9cf945be7c2cf88 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/54e2e6fa1f664f03a9cf945be7c2cf88 2023-03-27 04:58:03,970 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/54e2e6fa1f664f03a9cf945be7c2cf88, entries=150, sequenceid=102, filesize=11.6 K 2023-03-27 04:58:03,970 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/1f2e632d223f4ecdbcc2d7bd40012dad as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1f2e632d223f4ecdbcc2d7bd40012dad 2023-03-27 04:58:03,973 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1f2e632d223f4ecdbcc2d7bd40012dad, entries=150, sequenceid=102, filesize=11.6 K 2023-03-27 04:58:03,973 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 72e4f7cba88d8e5eeeccc51fc07b799f in 1237ms, sequenceid=102, compaction requested=true 2023-03-27 04:58:03,974 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:03,974 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:03,974 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:03,974 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:03,974 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,974 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:03,974 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:03,975 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,975 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:03,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:03,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:03,975 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:03,975 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:03,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:03,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:03,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:03,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:03,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:03,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@204e3975[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:03,976 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:03,976 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:03,976 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:03,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:03,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:03,977 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:03,977 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:03,977 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:03,977 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,977 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:03,977 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:03,977 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,977 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:03,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:03,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:03,977 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:03,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:03,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:03,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:03,979 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:03,979 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:03,979 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:03,979 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:03,979 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:03,980 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:03,980 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:03,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:03,980 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:03,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:03,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:03,981 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:03,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:03,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,983 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,983 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:03,983 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:03,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:03,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:03,983 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:03,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:03,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:03,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:03,983 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:03,983 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:03,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:03,983 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:03,983 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:03,983 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:03,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:03,983 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:03,983 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:03,983 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:04,083 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:04,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:04,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:04,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:04,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:04,086 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:04,086 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:04,086 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:04,086 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:04,086 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:04,086 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:04,087 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:04,087 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:04,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:04,087 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,087 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:04,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:04,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:04,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:04,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:04,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:04,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:04,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:04,088 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:04,088 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:04,088 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:04,088 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:04,088 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:04,088 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:04,088 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:04,088 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-27 04:58:04,088 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:04,088 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:04,088 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:04,088 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:04,088 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:04,088 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:04,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:04,238 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:04,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893144251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893144251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893144255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893144256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893144260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893144361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893144361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893144361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893144365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893144369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,496 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/e4ac6734295a45c398e6f945be2be5fb 2023-03-27 04:58:04,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893144568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893144569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893144569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893144570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893144575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893144876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893144877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893144877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893144878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:04,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893144882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:04,906 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/200b1ee7baa84e1d8788e114601ead9e 2023-03-27 04:58:05,314 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/1dc39bd6c94a4a50948a6bb4607a440c 2023-03-27 04:58:05,317 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/e4ac6734295a45c398e6f945be2be5fb as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/e4ac6734295a45c398e6f945be2be5fb 2023-03-27 04:58:05,320 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/e4ac6734295a45c398e6f945be2be5fb, entries=150, sequenceid=116, filesize=11.6 K 2023-03-27 04:58:05,320 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/200b1ee7baa84e1d8788e114601ead9e as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/200b1ee7baa84e1d8788e114601ead9e 2023-03-27 04:58:05,323 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/200b1ee7baa84e1d8788e114601ead9e, entries=150, sequenceid=116, filesize=11.6 K 2023-03-27 04:58:05,323 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/1dc39bd6c94a4a50948a6bb4607a440c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1dc39bd6c94a4a50948a6bb4607a440c 2023-03-27 04:58:05,326 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1dc39bd6c94a4a50948a6bb4607a440c, entries=150, sequenceid=116, filesize=11.6 K 2023-03-27 04:58:05,327 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 72e4f7cba88d8e5eeeccc51fc07b799f in 1239ms, sequenceid=116, compaction requested=true 2023-03-27 04:58:05,327 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:05,327 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:05,327 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:05,327 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:05,327 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,327 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:05,327 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:05,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:05,388 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-27 04:58:05,388 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:05,388 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:05,388 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:05,388 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:05,388 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:05,388 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:05,395 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/e4f756d520e94f56b294f4c9c8c2b3d2 2023-03-27 04:58:05,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:05,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893145399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:05,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893145403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,409 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,409 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,409 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,409 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:05,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:05,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d7cea3b[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:05,410 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:05,410 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,411 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,411 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:05,411 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:05,411 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:05,411 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,411 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:05,412 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:05,412 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,412 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,412 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:05,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893145404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:05,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:05,413 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:05,413 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:05,414 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:05,414 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:05,414 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:05,414 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,414 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:05,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,414 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:05,415 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:05,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893145405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,415 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:05,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893145406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:05,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,415 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,417 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,418 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:05,418 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,418 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:05,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:05,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:05,418 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:05,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:05,418 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,418 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:05,418 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,418 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,418 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:05,418 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,418 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:05,418 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:05,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:05,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893145509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:05,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893145509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:05,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893145513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,518 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:05,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:05,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:05,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:05,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:05,521 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:05,521 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:05,521 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:05,521 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:05,521 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,521 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:05,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893145515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:05,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893145516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,522 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,522 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:05,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:05,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:05,522 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,522 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:05,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:05,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:05,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:05,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:05,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:05,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:05,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:05,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:05,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:05,524 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:05,524 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:05,524 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:05,524 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:05,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:05,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:05,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:05,524 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:05,525 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,525 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,525 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,525 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:05,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:05,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:05,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@663603ae[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:05,526 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:05,526 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,527 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,527 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:05,527 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:05,527 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,527 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:05,527 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:05,528 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:05,528 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:05,528 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:05,529 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:05,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:05,531 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:05,531 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:05,531 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:05,531 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:05,531 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:05,532 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,532 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,532 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,532 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:05,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,534 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:05,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,536 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:05,536 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:05,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:05,536 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,536 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:05,536 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:05,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:05,536 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,536 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,536 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,537 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:05,537 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:05,537 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,537 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:05,637 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:05,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:05,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:05,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:05,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:05,639 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:05,639 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:05,639 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:05,639 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:05,640 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,640 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,640 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,640 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:05,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:05,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:05,641 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,641 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:05,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:05,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:05,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:05,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:05,642 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:05,642 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:05,642 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:05,642 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:05,642 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:05,642 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:05,642 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:05,642 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:05,642 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:05,642 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:05,642 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:05,642 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,642 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:05,642 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:05,643 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,643 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,643 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,643 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:05,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:05,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:05,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:05,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@741b7b56[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:05,644 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:05,644 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,645 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:05,645 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:05,645 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:05,645 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,645 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:05,646 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,646 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,646 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:05,646 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:05,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:05,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:05,647 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:05,647 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:05,647 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:05,647 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:05,647 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:05,648 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,648 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,648 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:05,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:05,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:05,648 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,649 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:05,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:05,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:05,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:05,651 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:05,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:05,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:05,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:05,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,651 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:05,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,651 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:05,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:05,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:05,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893145713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:05,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893145717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:05,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893145720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:05,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893145722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:05,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893145723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,751 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:05,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:05,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:05,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:05,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:05,754 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:05,754 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:05,754 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:05,754 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:05,754 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,754 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,755 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:05,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:05,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:05,755 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,755 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,755 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:05,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:05,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:05,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:05,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:05,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:05,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:05,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:05,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:05,756 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:05,756 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:05,756 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:05,756 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:05,757 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:05,757 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:05,757 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:05,757 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,757 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:05,757 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:05,757 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,757 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,758 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,758 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:05,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:05,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@254f6131[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:05,759 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:05,759 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,759 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,759 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:05,759 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:05,759 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:05,759 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,760 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:05,760 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,760 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:05,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,760 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:05,760 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:05,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:05,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:05,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:05,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:05,762 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:05,762 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:05,762 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:05,762 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:05,762 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:05,763 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,763 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,763 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:05,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:05,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:05,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,764 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:05,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,765 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:05,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,766 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:05,766 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:05,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:05,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:05,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:05,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:05,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,768 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:05,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:05,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:05,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:05,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,768 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,768 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:05,768 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:05,803 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/a5208cdf43804ced9e4bb95b496e5527 2023-03-27 04:58:05,868 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:05,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:05,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:05,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:05,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:05,872 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:05,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:05,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:05,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:05,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,872 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,873 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,873 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:05,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:05,873 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,873 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:05,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:05,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:05,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:05,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:05,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:05,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:05,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:05,874 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:05,874 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:05,874 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:05,874 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:05,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:05,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:05,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:05,874 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:05,875 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,875 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,875 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,875 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:05,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:05,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f486a91[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:05,876 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:05,876 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,878 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,878 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:05,878 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:05,878 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,878 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:05,878 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:05,879 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,879 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:05,879 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,879 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:05,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:05,881 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:05,881 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:05,881 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:05,881 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:05,881 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:05,882 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,882 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,882 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,883 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,884 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:05,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,886 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,886 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,886 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:05,886 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:05,886 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:05,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:05,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:05,886 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:05,886 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,886 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,886 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:05,887 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:05,887 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:05,887 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,987 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:05,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:05,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:05,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:05,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:05,990 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:05,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:05,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:05,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:05,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,990 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,991 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,991 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:05,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:05,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:05,991 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,991 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:05,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:05,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:05,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:05,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:05,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:05,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:05,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:05,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:05,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:05,992 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:05,992 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:05,992 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:05,992 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:05,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:05,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:05,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:05,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:05,993 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,993 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,993 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,993 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:05,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:05,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@593523a5[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:05,994 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:05,995 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,995 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:05,995 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:05,995 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:05,995 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,995 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:05,995 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:05,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:05,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:05,996 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:05,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:05,998 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:05,998 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:05,998 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:05,998 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:05,998 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:05,999 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,999 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:05,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:05,999 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:05,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:05,999 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:05,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:05,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:05,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,000 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:06,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,001 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,001 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,001 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:06,001 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,001 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,001 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,001 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,001 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:06,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:06,001 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:06,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:06,001 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:06,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:06,002 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:06,002 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:06,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:06,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:06,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893146020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893146025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893146027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893146030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893146033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,102 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:06,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:06,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:06,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:06,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:06,105 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:06,105 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:06,105 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:06,105 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:06,105 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,105 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,106 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:06,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:06,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:06,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:06,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:06,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:06,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:06,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:06,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:06,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:06,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:06,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:06,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:06,107 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:06,107 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:06,107 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:06,107 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:06,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:06,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:06,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:06,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:06,108 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,108 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:06,108 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,108 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:06,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:06,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:06,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:06,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@765b321e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:06,109 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:06,109 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,110 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,110 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:06,110 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:06,110 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:06,110 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,110 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:06,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:06,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:06,111 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:06,111 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:06,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:06,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:06,112 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:06,113 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:06,113 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:06,113 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:06,113 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:06,113 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,113 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:06,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:06,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,113 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:06,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,114 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,115 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:06,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,118 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,118 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:06,119 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,119 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:06,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:06,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:06,119 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:06,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:06,119 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:06,119 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:06,119 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:06,119 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,119 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:06,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:06,119 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,119 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,216 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/e187132e77004c73bb0d7e2332020247 2023-03-27 04:58:06,219 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:06,219 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/e4f756d520e94f56b294f4c9c8c2b3d2 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/e4f756d520e94f56b294f4c9c8c2b3d2 2023-03-27 04:58:06,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:06,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:06,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:06,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:06,222 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:06,222 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:06,222 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:06,222 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:06,222 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/e4f756d520e94f56b294f4c9c8c2b3d2, entries=200, sequenceid=135, filesize=14.0 K 2023-03-27 04:58:06,223 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,223 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,223 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/a5208cdf43804ced9e4bb95b496e5527 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/a5208cdf43804ced9e4bb95b496e5527 2023-03-27 04:58:06,223 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,223 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:06,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:06,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:06,223 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,224 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:06,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:06,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,224 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:06,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:06,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:06,225 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:06,225 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:06,225 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:06,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/a5208cdf43804ced9e4bb95b496e5527, entries=150, sequenceid=135, filesize=11.7 K 2023-03-27 04:58:06,229 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/e187132e77004c73bb0d7e2332020247 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/e187132e77004c73bb0d7e2332020247 2023-03-27 04:58:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:06,230 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:06,230 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:06,230 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:06,230 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:06,230 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:06,231 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/e187132e77004c73bb0d7e2332020247, entries=150, sequenceid=135, filesize=11.7 K 2023-03-27 04:58:06,232 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 72e4f7cba88d8e5eeeccc51fc07b799f in 844ms, sequenceid=135, compaction requested=true 2023-03-27 04:58:06,232 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:06,232 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:06,232 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-27 04:58:06,232 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-27 04:58:06,232 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:06,232 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:06,234 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 102258 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-27 04:58:06,234 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95188 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-27 04:58:06,234 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/A is initiating minor compaction (all files) 2023-03-27 04:58:06,234 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/B is initiating minor compaction (all files) 2023-03-27 04:58:06,234 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/A in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:06,234 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/B in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:06,234 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a53484e06a5046c389e6f0a46293d015, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a8c23eb358f84989a413ecd6143db071, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/70349533f17a4e21a6a6cfbd95336950, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/c3f800bdb4cd49688498e7b66aa8df00, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/ad12db80f0f948218ecf25bfee5ad2f0, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/5d4ef7d568d14467ab640dd05d9fc481, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/e4ac6734295a45c398e6f945be2be5fb, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/e4f756d520e94f56b294f4c9c8c2b3d2] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=99.9 K 2023-03-27 04:58:06,234 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/7d320b4903ed49b79c751e1cb7a5a995, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/e7889c114fd3434692dc5371c61ea742, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/082fc3a8016f44ad8708adaaf1b3cc02, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/815406931936497ea1f481d4389046f5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/c58379f2d71a4d6793318f94b8ec528f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/54e2e6fa1f664f03a9cf945be7c2cf88, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/200b1ee7baa84e1d8788e114601ead9e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/a5208cdf43804ced9e4bb95b496e5527] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=93.0 K 2023-03-27 04:58:06,234 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting a53484e06a5046c389e6f0a46293d015, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1679893078392 2023-03-27 04:58:06,234 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 7d320b4903ed49b79c751e1cb7a5a995, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1679893078392 2023-03-27 04:58:06,235 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting a8c23eb358f84989a413ecd6143db071, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1679893078415 2023-03-27 04:58:06,235 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 70349533f17a4e21a6a6cfbd95336950, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1679893078530 2023-03-27 04:58:06,235 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting e7889c114fd3434692dc5371c61ea742, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1679893078415 2023-03-27 04:58:06,235 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting c3f800bdb4cd49688498e7b66aa8df00, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679893079674 2023-03-27 04:58:06,235 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 082fc3a8016f44ad8708adaaf1b3cc02, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1679893078530 2023-03-27 04:58:06,236 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting ad12db80f0f948218ecf25bfee5ad2f0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1679893080811 2023-03-27 04:58:06,236 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 815406931936497ea1f481d4389046f5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679893079674 2023-03-27 04:58:06,236 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 5d4ef7d568d14467ab640dd05d9fc481, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1679893081959 2023-03-27 04:58:06,236 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting c58379f2d71a4d6793318f94b8ec528f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1679893080811 2023-03-27 04:58:06,236 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting e4ac6734295a45c398e6f945be2be5fb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1679893083101 2023-03-27 04:58:06,237 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 54e2e6fa1f664f03a9cf945be7c2cf88, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1679893081959 2023-03-27 04:58:06,237 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting e4f756d520e94f56b294f4c9c8c2b3d2, keycount=200, bloomtype=ROW, size=14.0 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679893084249 2023-03-27 04:58:06,237 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 200b1ee7baa84e1d8788e114601ead9e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1679893083101 2023-03-27 04:58:06,238 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting a5208cdf43804ced9e4bb95b496e5527, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679893084249 2023-03-27 04:58:06,252 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#A#compaction#389 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:06,253 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#B#compaction#388 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:06,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,254 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:06,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:06,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:06,254 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,254 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:06,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:06,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:06,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:06,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28c3930d[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:06,255 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:06,255 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,258 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:06,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:06,259 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,259 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:06,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:06,259 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:06,260 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:06,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:06,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:06,260 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:06,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:06,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:06,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:06,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:06,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:06,262 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:06,262 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:06,262 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:06,262 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:06,262 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:06,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:06,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:06,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:06,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:06,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:06,263 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:06,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:06,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,264 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:06,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,265 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:06,265 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:06,265 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:06,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:06,265 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:06,265 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:06,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:06,265 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,265 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,266 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,266 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,266 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:06,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:06,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:06,266 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:06,266 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:06,277 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/af06c26221754aa6a0cf7027358f9aac as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/af06c26221754aa6a0cf7027358f9aac 2023-03-27 04:58:06,280 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/B of 72e4f7cba88d8e5eeeccc51fc07b799f into af06c26221754aa6a0cf7027358f9aac(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:06,280 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:06,280 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/B, priority=8, startTime=1679893086232; duration=0sec 2023-03-27 04:58:06,281 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:06,281 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-27 04:58:06,282 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95188 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-27 04:58:06,282 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/C is initiating minor compaction (all files) 2023-03-27 04:58:06,282 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/C in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:06,283 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/168feefd03634291b02e629e9c8404d3, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/8927d160474246b7ad1c460ab26eab58, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/c8aad07a200847d592082c699cd4c101, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/45bf3280e00b4561ae8bfe123b5fedda, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/4f8894ce90034c14a1a7bb88dd8ff974, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1f2e632d223f4ecdbcc2d7bd40012dad, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1dc39bd6c94a4a50948a6bb4607a440c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/e187132e77004c73bb0d7e2332020247] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=93.0 K 2023-03-27 04:58:06,283 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 168feefd03634291b02e629e9c8404d3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1679893078392 2023-03-27 04:58:06,283 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 8927d160474246b7ad1c460ab26eab58, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1679893078415 2023-03-27 04:58:06,283 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting c8aad07a200847d592082c699cd4c101, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1679893078530 2023-03-27 04:58:06,283 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 45bf3280e00b4561ae8bfe123b5fedda, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679893079674 2023-03-27 04:58:06,284 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 4f8894ce90034c14a1a7bb88dd8ff974, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1679893080811 2023-03-27 04:58:06,284 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 1f2e632d223f4ecdbcc2d7bd40012dad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1679893081959 2023-03-27 04:58:06,284 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 1dc39bd6c94a4a50948a6bb4607a440c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1679893083101 2023-03-27 04:58:06,284 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting e187132e77004c73bb0d7e2332020247, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679893084249 2023-03-27 04:58:06,291 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#C#compaction#390 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:06,296 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/1391d8a0cbcf4efebb598e31a4e9e6c7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1391d8a0cbcf4efebb598e31a4e9e6c7 2023-03-27 04:58:06,299 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/C of 72e4f7cba88d8e5eeeccc51fc07b799f into 1391d8a0cbcf4efebb598e31a4e9e6c7(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:06,299 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:06,299 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/C, priority=8, startTime=1679893086232; duration=0sec 2023-03-27 04:58:06,299 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:06,366 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:06,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:06,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:06,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:06,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:06,369 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:06,369 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:06,369 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:06,369 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:06,369 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,369 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,372 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,372 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:06,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:06,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:06,372 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,372 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:06,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:06,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:06,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:06,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:06,373 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:06,373 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:06,373 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:06,373 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:06,373 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:06,373 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:06,373 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:06,373 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-27 04:58:06,374 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:06,374 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:06,374 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:06,374 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:06,374 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:06,375 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:06,382 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/fbff1d84370f40ec9fa60be20748287e 2023-03-27 04:58:06,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:06,550 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:06,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893146569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893146572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893146572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893146573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893146574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,678 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/c21425d556d4494c9a96059dbbeb4793 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/c21425d556d4494c9a96059dbbeb4793 2023-03-27 04:58:06,682 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/A of 72e4f7cba88d8e5eeeccc51fc07b799f into c21425d556d4494c9a96059dbbeb4793(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:06,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,682 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:06,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893146676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,682 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/A, priority=8, startTime=1679893086232; duration=0sec 2023-03-27 04:58:06,682 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:06,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893146679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893146679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893146683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893146684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,802 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/559a36d43a6a47f1881671cc519f659d 2023-03-27 04:58:06,809 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/10d42a4b3e7746e382933c0a4787dd4b 2023-03-27 04:58:06,812 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/fbff1d84370f40ec9fa60be20748287e as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/fbff1d84370f40ec9fa60be20748287e 2023-03-27 04:58:06,814 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/fbff1d84370f40ec9fa60be20748287e, entries=150, sequenceid=150, filesize=11.8 K 2023-03-27 04:58:06,815 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/559a36d43a6a47f1881671cc519f659d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/559a36d43a6a47f1881671cc519f659d 2023-03-27 04:58:06,818 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/559a36d43a6a47f1881671cc519f659d, entries=150, sequenceid=150, filesize=11.8 K 2023-03-27 04:58:06,818 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/10d42a4b3e7746e382933c0a4787dd4b as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/10d42a4b3e7746e382933c0a4787dd4b 2023-03-27 04:58:06,821 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/10d42a4b3e7746e382933c0a4787dd4b, entries=150, sequenceid=150, filesize=11.8 K 2023-03-27 04:58:06,821 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 72e4f7cba88d8e5eeeccc51fc07b799f in 448ms, sequenceid=150, compaction requested=false 2023-03-27 04:58:06,822 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:06,822 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:06,822 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:06,822 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:06,822 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,822 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:06,822 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:06,823 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,823 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:06,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:06,823 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,823 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:06,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:06,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:06,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:06,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64137866[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:06,824 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:06,824 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,825 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,825 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:06,825 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:06,825 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:06,825 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,825 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:06,826 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:06,826 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:06,826 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,826 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:06,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:06,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:06,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:06,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:06,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:06,828 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:06,828 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:06,828 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:06,828 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:06,828 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:06,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:06,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:06,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:06,829 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:06,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:06,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,831 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,833 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,833 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,833 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:06,833 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,833 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:06,833 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:06,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:06,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:06,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:06,833 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:06,833 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,833 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:06,833 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,833 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:06,833 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:06,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:06,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:06,887 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-27 04:58:06,888 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:06,888 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:06,888 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:06,888 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:06,888 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:06,888 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:06,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893146896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893146897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893146897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893146898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:06,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893146900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,933 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:06,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:06,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:06,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:06,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:06,935 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:06,935 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:06,936 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:06,936 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:06,936 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,936 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,937 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:06,937 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:06,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:06,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:06,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:06,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:06,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:06,938 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:06,938 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:06,938 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,938 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:06,938 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:06,938 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:06,938 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:06,938 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:06,938 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:06,938 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:06,938 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:06,938 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:06,938 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:06,939 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,939 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:06,939 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:06,939 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,939 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:06,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:06,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:06,940 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,940 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:06,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:06,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@144b574a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:06,941 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:06,941 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,942 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,942 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:06,942 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:06,942 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:06,942 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,942 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:06,943 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,943 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:06,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:06,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:06,943 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:06,943 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:06,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:06,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:06,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:06,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:06,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:06,945 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:06,945 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:06,945 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:06,945 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:06,945 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:06,945 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,945 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:06,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:06,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,945 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:06,946 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:06,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:06,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:06,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:06,946 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:06,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:06,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:06,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:06,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:06,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:06,948 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:06,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:06,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:06,948 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:06,948 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:06,949 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:06,949 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:06,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:06,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:06,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:06,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:06,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:07,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:07,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893147001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:07,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893147001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:07,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893147001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:07,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893147002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:07,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893147002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,048 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:07,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:07,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:07,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:07,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:07,051 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:07,051 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:07,051 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:07,051 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:07,051 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,051 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,052 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:07,052 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:07,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:07,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:07,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:07,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:07,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:07,053 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:07,053 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:07,053 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:07,054 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,054 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:07,059 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:07,059 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:07,059 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:07,059 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:07,059 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:07,059 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:07,059 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:07,059 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:07,059 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,059 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:07,059 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:07,067 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,067 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:07,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:07,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b2f5c1f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:07,069 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:07,069 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,069 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,070 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:07,070 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:07,070 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:07,070 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,070 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:07,070 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:07,070 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:07,070 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,070 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:07,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:07,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:07,072 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:07,072 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:07,072 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:07,072 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:07,072 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:07,073 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,073 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,073 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:07,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:07,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:07,074 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,075 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:07,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:07,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:07,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:07,076 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:07,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:07,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:07,076 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:07,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,076 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:07,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:07,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:07,176 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:07,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:07,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:07,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:07,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:07,179 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:07,179 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:07,179 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:07,179 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:07,179 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,179 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,180 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:07,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:07,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:07,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:07,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:07,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:07,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:07,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:07,192 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:07,192 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:07,192 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:07,193 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:07,193 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:07,193 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:07,193 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:07,193 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:07,193 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:07,193 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:07,193 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:07,194 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,194 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:07,194 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:07,195 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,195 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,195 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,196 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:07,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fc0eb8c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:07,197 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:07,197 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,198 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:07,198 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:07,198 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:07,198 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,199 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:07,199 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:07,199 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:07,199 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,199 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:07,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:07,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:07,201 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:07,201 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:07,201 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:07,201 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:07,201 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:07,202 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,202 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,202 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:07,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:07,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,203 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,205 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,205 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:07,205 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:07,205 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:07,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:07,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:07,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:07,205 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,205 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:07,206 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,206 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:07,206 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:07,206 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:07,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:07,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:07,206 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,206 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893147203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:07,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893147207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:07,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893147207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:07,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893147207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:07,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893147207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,296 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/a6a515094d1244b89febb942c6852182 2023-03-27 04:58:07,306 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:07,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:07,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:07,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:07,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:07,308 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:07,308 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:07,308 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:07,308 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:07,309 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,309 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,309 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,309 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:07,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:07,310 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,310 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:07,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:07,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:07,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:07,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:07,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:07,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:07,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:07,311 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:07,311 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:07,311 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:07,311 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:07,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:07,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:07,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:07,311 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:07,312 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,312 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,312 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,312 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:07,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:07,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ef725a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:07,313 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:07,313 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,313 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,313 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:07,313 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:07,313 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,313 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:07,313 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:07,314 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,314 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:07,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,314 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:07,314 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:07,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:07,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:07,316 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:07,316 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:07,316 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:07,316 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:07,316 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:07,317 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,317 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,317 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:07,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:07,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:07,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,318 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,319 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,320 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:07,320 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,320 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,320 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,320 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,320 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:07,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:07,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:07,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:07,320 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,320 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,320 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:07,320 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:07,321 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:07,320 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:07,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:07,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:07,421 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:07,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:07,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:07,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:07,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:07,423 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:07,423 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:07,423 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:07,423 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:07,424 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,424 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,424 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,424 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:07,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:07,424 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,424 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:07,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:07,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:07,425 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:07,425 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:07,425 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:07,426 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:07,426 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:07,426 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:07,426 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:07,426 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:07,426 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:07,426 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:07,426 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:07,426 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,426 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:07,426 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:07,427 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:07,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:07,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e509a0a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:07,429 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:07,429 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,429 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:07,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:07,429 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:07,430 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,430 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:07,431 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,431 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,431 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:07,431 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:07,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:07,432 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:07,432 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:07,432 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:07,432 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:07,433 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:07,433 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,433 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:07,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,433 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,434 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,435 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,436 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,436 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,436 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,436 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:07,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:07,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:07,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:07,436 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:07,436 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:07,436 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,437 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:07,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:07,436 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:07,437 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:07,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:07,437 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,437 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:07,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893147508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:07,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893147509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:07,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893147510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:07,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893147510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:07,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893147511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,537 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:07,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:07,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:07,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:07,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:07,539 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:07,539 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:07,540 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:07,540 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:07,540 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,540 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,540 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:07,540 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:07,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:07,541 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,541 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:07,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:07,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:07,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:07,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:07,542 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:07,542 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:07,542 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:07,542 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:07,542 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:07,542 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:07,542 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:07,542 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:07,542 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:07,542 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:07,542 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:07,542 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,542 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:07,542 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:07,543 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,543 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,544 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,544 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:07,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:07,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f5ee67b[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:07,544 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:07,544 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,545 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,545 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:07,545 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:07,545 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:07,545 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,545 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:07,545 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:07,545 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,545 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:07,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,545 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:07,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:07,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:07,548 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:07,548 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:07,548 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:07,548 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:07,548 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:07,549 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,549 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,549 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,549 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,550 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,551 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:07,552 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:07,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:07,552 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,552 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:07,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:07,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:07,552 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,552 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:07,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:07,552 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,552 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,552 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,552 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,552 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:07,552 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:07,652 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:07,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:07,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:07,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:07,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:07,655 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:07,655 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:07,655 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:07,655 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:07,655 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,655 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,656 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,656 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:07,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:07,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:07,656 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,656 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:07,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:07,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:07,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:07,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:07,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:07,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:07,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:07,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:07,658 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:07,658 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:07,658 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:07,659 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:07,659 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:07,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:07,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:07,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:07,659 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:07,660 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,660 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,660 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,660 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:07,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:07,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77cafe97[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:07,661 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:07,661 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,661 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:07,662 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:07,662 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,662 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:07,662 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:07,662 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,662 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:07,662 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:07,663 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:07,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:07,664 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:07,664 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:07,665 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:07,665 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:07,665 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:07,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,666 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,667 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,668 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:07,668 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:07,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:07,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:07,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:07,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:07,668 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,668 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,668 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:07,669 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,669 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,669 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:07,669 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,669 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:07,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:07,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:07,669 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,669 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,704 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/42f23daf5fc8477facf527608d651db4 2023-03-27 04:58:07,769 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:07,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:07,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:07,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:07,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:07,771 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:07,771 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:07,771 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:07,771 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:07,772 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,772 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,772 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:07,772 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:07,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:07,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,773 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,773 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:07,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:07,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:07,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:07,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:07,774 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:07,774 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:07,774 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:07,774 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:07,774 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:07,774 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:07,774 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:07,774 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:07,774 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:07,774 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:07,774 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:07,774 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,774 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:07,774 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:07,775 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,775 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,775 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,775 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:07,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:07,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c3c2f96[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:07,777 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:07,777 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,777 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,777 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:07,777 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:07,777 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,778 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:07,778 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:07,778 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:07,778 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:07,778 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,779 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:07,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:07,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:07,781 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:07,781 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:07,781 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:07,781 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:07,781 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:07,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:07,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:07,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:07,782 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,783 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,785 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,785 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,785 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,785 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:07,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:07,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:07,785 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:07,785 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:07,785 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:07,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:07,785 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:07,785 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:07,785 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:07,785 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,785 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:07,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:07,894 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:07,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:07,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:07,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:07,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:07,896 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:07,896 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:07,896 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:07,896 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:07,897 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,897 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,897 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:07,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:07,897 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:07,898 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,898 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:07,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:07,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:07,898 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:07,898 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:07,898 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:07,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:07,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:07,899 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:07,899 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:07,899 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:07,899 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:07,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:07,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:07,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:07,899 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:07,900 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,900 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,900 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:07,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:07,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46867ad1[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:07,901 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:07,901 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,901 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,901 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,902 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:07,901 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:07,902 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:07,902 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:07,902 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:07,902 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:07,902 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,902 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:07,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:07,905 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:07,905 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:07,905 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:07,905 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:07,905 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:07,907 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,907 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,907 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:07,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:07,907 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:07,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:07,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,908 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:07,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:07,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,910 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,910 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:07,910 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,910 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:07,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:07,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:07,910 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:07,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:07,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:07,910 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:07,910 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:07,910 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,910 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:07,910 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:07,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:07,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:07,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:07,910 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:07,910 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:08,011 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:08,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:08,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:08,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:08,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:08,013 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:08,013 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:08,013 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:08,013 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:08,014 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:08,014 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:08,014 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:08,014 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:08,015 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,015 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:08,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:08,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:08,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:08,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893148013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:08,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893148015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:08,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:08,017 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:08,017 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:08,017 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:08,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:08,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:08,019 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:08,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893148015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,019 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:08,019 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:08,019 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:08,020 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:08,020 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:08,020 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:08,020 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,020 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:08,020 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:08,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:08,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893148017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:08,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893148017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,034 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:08,034 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:08,034 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:08,034 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:08,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:08,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e633144[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:08,035 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:08,035 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:08,042 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:08,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:08,042 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:08,043 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:08,043 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:08,050 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:08,050 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,050 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:08,050 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:08,051 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:08,051 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:08,051 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:08,051 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:08,051 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:08,051 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:08,051 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:08,052 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:08,052 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:08,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:08,052 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:08,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:08,052 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:08,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:08,053 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:08,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:08,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:08,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,054 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,054 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:08,054 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:08,054 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:08,054 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:08,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:08,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:08,054 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:08,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:08,054 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:08,054 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:08,055 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:08,055 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:08,055 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:08,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:08,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:08,055 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:08,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:08,112 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/c28e24c316924de89d91425bcfaba933 2023-03-27 04:58:08,115 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/a6a515094d1244b89febb942c6852182 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a6a515094d1244b89febb942c6852182 2023-03-27 04:58:08,118 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a6a515094d1244b89febb942c6852182, entries=250, sequenceid=172, filesize=16.4 K 2023-03-27 04:58:08,119 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/42f23daf5fc8477facf527608d651db4 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/42f23daf5fc8477facf527608d651db4 2023-03-27 04:58:08,121 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/42f23daf5fc8477facf527608d651db4, entries=150, sequenceid=172, filesize=11.8 K 2023-03-27 04:58:08,122 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/c28e24c316924de89d91425bcfaba933 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/c28e24c316924de89d91425bcfaba933 2023-03-27 04:58:08,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/c28e24c316924de89d91425bcfaba933, entries=150, sequenceid=172, filesize=11.8 K 2023-03-27 04:58:08,125 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 72e4f7cba88d8e5eeeccc51fc07b799f in 1238ms, sequenceid=172, compaction requested=true 2023-03-27 04:58:08,125 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:08,125 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:08,125 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:08,125 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:08,125 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:08,125 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:08,126 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41112 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:08,126 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/A is initiating minor compaction (all files) 2023-03-27 04:58:08,126 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36332 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:08,126 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/A in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:08,126 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/B is initiating minor compaction (all files) 2023-03-27 04:58:08,126 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/c21425d556d4494c9a96059dbbeb4793, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/fbff1d84370f40ec9fa60be20748287e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a6a515094d1244b89febb942c6852182] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=40.1 K 2023-03-27 04:58:08,126 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/B in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:08,126 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/af06c26221754aa6a0cf7027358f9aac, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/559a36d43a6a47f1881671cc519f659d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/42f23daf5fc8477facf527608d651db4] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=35.5 K 2023-03-27 04:58:08,126 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting c21425d556d4494c9a96059dbbeb4793, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679893084249 2023-03-27 04:58:08,127 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting af06c26221754aa6a0cf7027358f9aac, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679893084249 2023-03-27 04:58:08,127 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting fbff1d84370f40ec9fa60be20748287e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1679893085398 2023-03-27 04:58:08,127 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 559a36d43a6a47f1881671cc519f659d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1679893085398 2023-03-27 04:58:08,127 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting a6a515094d1244b89febb942c6852182, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1679893086571 2023-03-27 04:58:08,127 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 42f23daf5fc8477facf527608d651db4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1679893086571 2023-03-27 04:58:08,132 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#A#compaction#397 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:08,133 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#B#compaction#398 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:08,155 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:08,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:08,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:08,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:08,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:08,157 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:08,157 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:08,157 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:08,157 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:08,158 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:08,158 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:08,158 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:08,158 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:08,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:08,159 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:08,159 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:08,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:08,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:08,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:08,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:08,159 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:08,159 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:08,159 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:08,160 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:08,160 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:08,160 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:08,160 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:08,160 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-27 04:58:08,160 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:08,160 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:08,160 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:08,160 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:08,160 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:08,160 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:08,168 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/dddf239c70ba4ccd864657693462e2c5 2023-03-27 04:58:08,538 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/05a0d2c8a35b49e8b6805dd8b370b461 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/05a0d2c8a35b49e8b6805dd8b370b461 2023-03-27 04:58:08,541 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/A of 72e4f7cba88d8e5eeeccc51fc07b799f into 05a0d2c8a35b49e8b6805dd8b370b461(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:08,541 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:08,541 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/A, priority=13, startTime=1679893088125; duration=0sec 2023-03-27 04:58:08,541 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:08,541 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:08,542 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36332 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:08,542 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/C is initiating minor compaction (all files) 2023-03-27 04:58:08,543 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/C in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:08,543 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1391d8a0cbcf4efebb598e31a4e9e6c7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/10d42a4b3e7746e382933c0a4787dd4b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/c28e24c316924de89d91425bcfaba933] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=35.5 K 2023-03-27 04:58:08,543 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 1391d8a0cbcf4efebb598e31a4e9e6c7, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679893084249 2023-03-27 04:58:08,543 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 10d42a4b3e7746e382933c0a4787dd4b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1679893085398 2023-03-27 04:58:08,544 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting c28e24c316924de89d91425bcfaba933, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1679893086571 2023-03-27 04:58:08,544 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/5a90307443f94509a1034861c90ffad3 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/5a90307443f94509a1034861c90ffad3 2023-03-27 04:58:08,547 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/B of 72e4f7cba88d8e5eeeccc51fc07b799f into 5a90307443f94509a1034861c90ffad3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:08,547 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:08,547 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/B, priority=13, startTime=1679893088125; duration=0sec 2023-03-27 04:58:08,547 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:08,548 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#C#compaction#401 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:08,579 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/f20ec49f44e042389f76a4328ebf2e8c 2023-03-27 04:58:08,955 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/3c4f6f6839f34de18b3b0be7ed9dce1d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/3c4f6f6839f34de18b3b0be7ed9dce1d 2023-03-27 04:58:08,958 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/C of 72e4f7cba88d8e5eeeccc51fc07b799f into 3c4f6f6839f34de18b3b0be7ed9dce1d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:08,958 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:08,958 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/C, priority=13, startTime=1679893088125; duration=0sec 2023-03-27 04:58:08,958 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:08,993 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/5b9398f767a949b4966b7705bb7c6cd0 2023-03-27 04:58:08,996 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/dddf239c70ba4ccd864657693462e2c5 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/dddf239c70ba4ccd864657693462e2c5 2023-03-27 04:58:08,998 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/dddf239c70ba4ccd864657693462e2c5, entries=150, sequenceid=183, filesize=11.8 K 2023-03-27 04:58:08,999 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/f20ec49f44e042389f76a4328ebf2e8c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/f20ec49f44e042389f76a4328ebf2e8c 2023-03-27 04:58:09,001 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/f20ec49f44e042389f76a4328ebf2e8c, entries=150, sequenceid=183, filesize=11.8 K 2023-03-27 04:58:09,002 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/5b9398f767a949b4966b7705bb7c6cd0 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/5b9398f767a949b4966b7705bb7c6cd0 2023-03-27 04:58:09,005 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/5b9398f767a949b4966b7705bb7c6cd0, entries=150, sequenceid=183, filesize=11.8 K 2023-03-27 04:58:09,005 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 72e4f7cba88d8e5eeeccc51fc07b799f in 845ms, sequenceid=183, compaction requested=false 2023-03-27 04:58:09,005 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:09,005 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:09,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:09,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:09,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:09,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:09,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,006 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,007 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,007 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:09,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:09,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e7d4059[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:09,008 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:09,008 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,009 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,009 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:09,009 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,009 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:09,009 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:09,009 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:09,009 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,009 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:09,010 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,010 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:09,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:09,011 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:09,012 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:09,012 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:09,012 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:09,012 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:09,013 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,013 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,013 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,013 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,014 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,015 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,015 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:09,016 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:09,016 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:09,016 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:09,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:09,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:09,016 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,016 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,016 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,016 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:09,016 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,016 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:09,016 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:09,027 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-27 04:58:09,028 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:09,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:09,028 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:09,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:09,028 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:09,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:09,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893149049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893149053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893089042, max timestamp=1679893089054Num uniques 859; 2023-03-27 04:58:09,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893089042, max timestamp=1679893089054Num uniques 859; 2023-03-27 04:58:09,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:58:09,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893089042, max timestamp=1679893089054Num uniques 859; 2023-03-27 04:58:09,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:58:09,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:58:09,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893149083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893149086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893149088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,116 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:09,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:09,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:09,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:09,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:09,119 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:09,119 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:09,119 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:09,119 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:09,119 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,119 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,120 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:09,120 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:09,120 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,120 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:09,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:09,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:09,121 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:09,121 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:09,121 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:09,121 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:09,121 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:09,121 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:09,121 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:09,121 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:09,121 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:09,121 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:09,121 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:09,121 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,121 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:09,121 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:09,122 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,122 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,122 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,122 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:09,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7320929f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:09,123 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:09,123 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,124 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,124 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:09,124 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,124 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:09,124 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:09,124 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:09,125 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:09,125 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,125 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,125 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:09,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:09,127 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:09,127 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:09,127 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:09,127 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:09,127 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:09,128 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,128 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,128 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,129 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,130 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,132 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:09,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:09,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:09,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:09,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:09,133 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:09,133 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:09,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,133 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,133 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,133 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,133 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893149156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893149157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893149194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893149194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893149194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,233 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:09,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:09,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:09,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:09,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:09,235 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:09,235 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:09,235 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:09,235 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:09,235 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,235 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,236 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,236 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:09,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:09,236 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,236 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:09,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:09,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:09,239 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:09,239 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:09,239 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:09,239 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:09,239 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:09,239 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:09,239 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:09,239 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:09,239 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:09,239 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:09,239 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:09,239 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,239 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:09,239 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:09,240 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,240 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,240 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,240 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:09,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6853ed8[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:09,241 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:09,241 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,242 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,242 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:09,242 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:09,242 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,242 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:09,242 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:09,243 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,243 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:09,243 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,243 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:09,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:09,244 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:09,244 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:09,244 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:09,244 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:09,244 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:09,245 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,245 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,245 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,245 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,246 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,248 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,248 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:09,248 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,248 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,248 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:09,248 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,248 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:09,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:09,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:09,248 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:09,248 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,248 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,248 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:09,248 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:09,348 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:09,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:09,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:09,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:09,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:09,351 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:09,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:09,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:09,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:09,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,352 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:09,352 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:09,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:09,352 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,352 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:09,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:09,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:09,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:09,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:09,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:09,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:09,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:09,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:09,353 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:09,353 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:09,353 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:09,353 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:09,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:09,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:09,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:09,353 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:09,354 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,354 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,354 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,355 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:09,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@513bd65d[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:09,356 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:09,357 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,357 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,357 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:09,357 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:09,357 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:09,357 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,357 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:09,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,358 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:09,358 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,358 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:09,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:09,360 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:09,360 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:09,360 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:09,360 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:09,360 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:09,361 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,361 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,361 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,363 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,364 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:09,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:09,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:09,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:09,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,365 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:09,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,366 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,366 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,366 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,366 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,366 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:09,366 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:09,366 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893149363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893149364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893149397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893149398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893149400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,438 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/d391527dd9ac4f0c80a6dc0a5308212a 2023-03-27 04:58:09,466 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:09,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:09,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:09,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:09,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:09,469 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:09,469 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:09,469 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:09,469 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:09,469 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,469 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,470 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:09,470 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:09,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:09,470 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,470 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:09,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:09,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:09,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:09,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:09,471 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:09,471 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:09,471 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:09,471 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:09,471 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:09,471 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:09,471 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:09,471 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:09,471 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:09,471 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:09,471 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:09,471 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,471 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:09,471 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:09,472 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,472 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,473 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,473 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:09,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55c03bab[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:09,473 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:09,474 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,474 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,474 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:09,474 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:09,474 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:09,474 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,474 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:09,475 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:09,475 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,475 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:09,475 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:09,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:09,477 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:09,477 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:09,477 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:09,477 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:09,477 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:09,478 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,478 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,478 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,478 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,479 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,481 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,481 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,481 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,481 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,481 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,481 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:09,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:09,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:09,481 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,481 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,482 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:09,482 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:09,482 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:09,482 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:09,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,582 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:09,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:09,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:09,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:09,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:09,584 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:09,584 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:09,584 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:09,584 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:09,585 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,585 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,585 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,585 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:09,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:09,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:09,585 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,585 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:09,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:09,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:09,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:09,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:09,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:09,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:09,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:09,586 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:09,586 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:09,586 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:09,586 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:09,587 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:09,587 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:09,587 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:09,587 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,587 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:09,587 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:09,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,587 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,588 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,588 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:09,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2203efe6[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:09,589 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:09,590 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,590 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,590 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:09,590 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:09,590 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,590 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:09,590 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:09,591 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:09,591 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,591 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,591 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:09,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:09,592 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:09,593 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:09,593 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:09,593 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:09,593 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:09,593 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,593 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,593 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,594 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,594 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,596 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:09,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:09,596 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:09,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:09,596 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,596 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,596 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,596 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:09,596 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,596 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:09,596 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,596 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:09,596 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,596 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893149669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893149669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,696 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:09,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:09,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:09,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:09,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:09,699 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:09,699 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:09,699 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:09,699 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:09,699 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,699 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,700 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,700 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:09,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:09,700 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,700 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:09,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:09,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:09,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:09,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:09,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:09,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:09,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:09,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:09,704 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:09,704 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:09,704 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:09,704 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:09,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:09,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:09,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:09,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:09,705 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,705 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,705 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,705 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:09,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893149702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:09,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ddf6680[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:09,707 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:09,707 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893149703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:09,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893149703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,707 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,707 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:09,707 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:09,708 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:09,708 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,708 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:09,708 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:09,708 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,708 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,708 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:09,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:09,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:09,710 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:09,710 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:09,710 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:09,710 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:09,710 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:09,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,711 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,712 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,713 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,713 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,713 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,713 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:09,714 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,714 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:09,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:09,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:09,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:09,714 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,714 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,714 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,714 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:09,714 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:09,714 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:09,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,814 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:09,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:09,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:09,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:09,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:09,817 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:09,817 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:09,817 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:09,817 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:09,817 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,817 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,818 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:09,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:09,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:09,818 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,818 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,818 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:09,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:09,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:09,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:09,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:09,819 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:09,819 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:09,819 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:09,820 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:09,820 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:09,820 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:09,820 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:09,820 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:09,820 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:09,820 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:09,820 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:09,820 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,820 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:09,820 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:09,821 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,821 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,821 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,821 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:09,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:09,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@259c058[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:09,822 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:09,822 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,823 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,823 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:09,823 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:09,823 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:09,823 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,823 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:09,824 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,824 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:09,824 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,824 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:09,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:09,825 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:09,825 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:09,825 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:09,825 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:09,825 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:09,826 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,826 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,826 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,827 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,828 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,830 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,830 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,830 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,830 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,830 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,830 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:09,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:09,830 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:09,830 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:09,830 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:09,830 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:09,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/2c3b9abc92ec4b248e25cef9f683e53e 2023-03-27 04:58:09,930 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:09,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:09,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:09,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:09,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:09,933 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:09,933 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:09,933 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:09,933 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:09,933 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,933 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,934 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:09,934 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:09,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:09,934 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,934 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:09,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:09,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:09,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:09,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:09,935 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:09,935 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:09,935 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:09,935 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:09,936 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:09,936 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:09,936 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:09,936 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:09,936 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:09,936 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:09,936 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:09,936 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,936 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:09,936 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:09,937 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:09,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:09,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5610aedb[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:09,938 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:09,939 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,939 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,939 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:09,939 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,939 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:09,939 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:09,940 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:09,940 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:09,940 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,940 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,940 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:09,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:09,942 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:09,942 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:09,942 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:09,942 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:09,942 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:09,943 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,943 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:09,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,943 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,943 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:09,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:09,944 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,945 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,945 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:09,945 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:09,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:09,946 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:09,946 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:09,946 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:09,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:09,946 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:09,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:09,946 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,946 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:09,946 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:09,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:09,946 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:09,946 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:10,046 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:10,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:10,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:10,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:10,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:10,049 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:10,049 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:10,049 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:10,049 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:10,049 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,049 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:10,050 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:10,050 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:10,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:10,050 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,050 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:10,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:10,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:10,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:10,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:10,051 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:10,051 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:10,051 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:10,051 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:10,051 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:10,051 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:10,051 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:10,051 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:10,051 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:10,052 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:10,052 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:10,052 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,052 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:10,052 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:10,052 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,052 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:10,053 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:10,053 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:10,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:10,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b511720[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:10,054 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:10,054 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:10,054 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:10,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:10,054 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:10,054 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,054 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:10,054 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:10,055 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:10,055 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,055 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:10,055 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:10,055 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:10,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:10,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:10,057 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:10,057 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:10,057 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:10,057 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:10,057 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:10,058 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,058 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:10,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,058 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:10,058 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:10,059 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:10,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:10,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,060 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:10,060 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:10,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:10,060 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:10,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:10,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:10,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:10,060 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,061 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:10,061 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:10,061 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,061 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:10,061 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:10,061 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:10,061 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:10,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:10,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:10,061 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:10,161 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:10,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:10,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:10,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:10,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:10,164 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:10,164 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:10,164 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:10,164 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:10,164 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,164 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:10,165 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,165 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:10,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:10,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:10,165 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,165 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:10,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:10,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:10,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:10,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:10,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:10,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:10,166 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:10,166 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:10,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:10,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:10,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:10,167 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:10,167 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:10,167 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:10,167 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:10,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:10,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:10,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:10,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:10,168 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:10,168 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:10,168 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:10,168 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:10,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:10,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:10,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:10,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dcc87ac[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:10,169 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:10,169 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:10,170 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:10,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:10,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:10,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:10,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:10,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:10,170 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,170 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:10,170 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:10,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:10,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:10,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:10,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:10,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:10,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:10,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:10,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:10,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:10,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:10,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:10,172 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:10,172 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:10,172 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:10,173 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:10,173 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:10,173 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,173 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,173 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:10,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:10,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:10,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:10,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:10,174 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:10,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:10,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:10,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:10,175 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:10,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:10,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,177 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,177 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:10,177 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:10,177 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:10,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:10,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:10,177 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:10,177 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:10,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:10,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:10,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:10,177 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:10,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:10,177 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:10,177 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:10,177 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:10,177 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:10,177 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:10,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43148 deadline: 1679893150182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:10,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43118 deadline: 1679893150182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:10,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:10,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893150208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43142 deadline: 1679893150208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:10,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43130 deadline: 1679893150211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,254 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/81c694c12a364045bfeb7adc470575f6 2023-03-27 04:58:10,257 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/d391527dd9ac4f0c80a6dc0a5308212a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/d391527dd9ac4f0c80a6dc0a5308212a 2023-03-27 04:58:10,260 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/d391527dd9ac4f0c80a6dc0a5308212a, entries=200, sequenceid=198, filesize=14.1 K 2023-03-27 04:58:10,260 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/2c3b9abc92ec4b248e25cef9f683e53e as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/2c3b9abc92ec4b248e25cef9f683e53e 2023-03-27 04:58:10,262 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/2c3b9abc92ec4b248e25cef9f683e53e, entries=150, sequenceid=198, filesize=11.8 K 2023-03-27 04:58:10,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/81c694c12a364045bfeb7adc470575f6 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/81c694c12a364045bfeb7adc470575f6 2023-03-27 04:58:10,265 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/81c694c12a364045bfeb7adc470575f6, entries=150, sequenceid=198, filesize=11.8 K 2023-03-27 04:58:10,266 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 72e4f7cba88d8e5eeeccc51fc07b799f in 1239ms, sequenceid=198, compaction requested=true 2023-03-27 04:58:10,266 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:10,266 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:10,266 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:10,266 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:10,266 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:10,266 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:10,267 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38874 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:10,267 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:10,267 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/A is initiating minor compaction (all files) 2023-03-27 04:58:10,267 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/B is initiating minor compaction (all files) 2023-03-27 04:58:10,268 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/A in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:10,268 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/B in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:10,268 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/05a0d2c8a35b49e8b6805dd8b370b461, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/dddf239c70ba4ccd864657693462e2c5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/d391527dd9ac4f0c80a6dc0a5308212a] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=38.0 K 2023-03-27 04:58:10,268 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/5a90307443f94509a1034861c90ffad3, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/f20ec49f44e042389f76a4328ebf2e8c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/2c3b9abc92ec4b248e25cef9f683e53e] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=35.6 K 2023-03-27 04:58:10,268 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 05a0d2c8a35b49e8b6805dd8b370b461, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1679893086571 2023-03-27 04:58:10,268 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 5a90307443f94509a1034861c90ffad3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1679893086571 2023-03-27 04:58:10,268 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting dddf239c70ba4ccd864657693462e2c5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679893086894 2023-03-27 04:58:10,268 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting f20ec49f44e042389f76a4328ebf2e8c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679893086894 2023-03-27 04:58:10,268 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting d391527dd9ac4f0c80a6dc0a5308212a, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679893089026 2023-03-27 04:58:10,268 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 2c3b9abc92ec4b248e25cef9f683e53e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679893089026 2023-03-27 04:58:10,273 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#A#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:10,274 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#B#compaction#407 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:10,277 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:10,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:10,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:10,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:10,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:10,282 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:10,282 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:10,282 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:10,282 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:10,283 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,283 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:10,283 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,283 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:10,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:10,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:10,283 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:10,284 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:10,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:10,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:10,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:10,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:10,285 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:10,285 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:10,285 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:10,285 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:10,285 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:10,285 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:10,285 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:10,285 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-03-27 04:58:10,285 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:10,285 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:10,285 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:10,285 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:10,285 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:10,285 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:10,291 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/695b1353189a4b039c2f6d0579d03251 2023-03-27 04:58:10,683 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/87c00c1c6e564e7ca5a6867820cf5f08 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/87c00c1c6e564e7ca5a6867820cf5f08 2023-03-27 04:58:10,686 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/A of 72e4f7cba88d8e5eeeccc51fc07b799f into 87c00c1c6e564e7ca5a6867820cf5f08(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:10,686 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:10,686 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/A, priority=13, startTime=1679893090266; duration=0sec 2023-03-27 04:58:10,686 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:10,686 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:10,687 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:10,687 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/C is initiating minor compaction (all files) 2023-03-27 04:58:10,687 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/C in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:10,687 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/3c4f6f6839f34de18b3b0be7ed9dce1d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/5b9398f767a949b4966b7705bb7c6cd0, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/81c694c12a364045bfeb7adc470575f6] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=35.6 K 2023-03-27 04:58:10,688 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 3c4f6f6839f34de18b3b0be7ed9dce1d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1679893086571 2023-03-27 04:58:10,688 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 5b9398f767a949b4966b7705bb7c6cd0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679893086894 2023-03-27 04:58:10,689 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 81c694c12a364045bfeb7adc470575f6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679893089026 2023-03-27 04:58:10,690 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/bab189c8c3264e71be3ea9cc51afdfb6 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/bab189c8c3264e71be3ea9cc51afdfb6 2023-03-27 04:58:10,694 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#C#compaction#410 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:10,695 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/B of 72e4f7cba88d8e5eeeccc51fc07b799f into bab189c8c3264e71be3ea9cc51afdfb6(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:10,695 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:10,695 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/B, priority=13, startTime=1679893090266; duration=0sec 2023-03-27 04:58:10,695 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:10,699 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/8e2c756acdcc4e3d93bfbeeef1eff684 2023-03-27 04:58:11,102 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/f496c7da19864989ba87741d50697355 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/f496c7da19864989ba87741d50697355 2023-03-27 04:58:11,105 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/C of 72e4f7cba88d8e5eeeccc51fc07b799f into f496c7da19864989ba87741d50697355(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:11,105 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:11,105 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/C, priority=13, startTime=1679893090266; duration=0sec 2023-03-27 04:58:11,105 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:11,106 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/da4c427c71084eecbc33a1b70f8f9d5f 2023-03-27 04:58:11,109 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/695b1353189a4b039c2f6d0579d03251 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/695b1353189a4b039c2f6d0579d03251 2023-03-27 04:58:11,112 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/695b1353189a4b039c2f6d0579d03251, entries=150, sequenceid=232, filesize=11.8 K 2023-03-27 04:58:11,112 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/8e2c756acdcc4e3d93bfbeeef1eff684 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/8e2c756acdcc4e3d93bfbeeef1eff684 2023-03-27 04:58:11,115 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/8e2c756acdcc4e3d93bfbeeef1eff684, entries=150, sequenceid=232, filesize=11.8 K 2023-03-27 04:58:11,116 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/da4c427c71084eecbc33a1b70f8f9d5f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/da4c427c71084eecbc33a1b70f8f9d5f 2023-03-27 04:58:11,119 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/da4c427c71084eecbc33a1b70f8f9d5f, entries=150, sequenceid=232, filesize=11.8 K 2023-03-27 04:58:11,119 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=0 B/0 for 72e4f7cba88d8e5eeeccc51fc07b799f in 834ms, sequenceid=232, compaction requested=false 2023-03-27 04:58:11,119 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:11,119 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:11,119 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:11,119 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:11,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:11,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:11,120 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,121 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,121 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:11,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51dbcf4d[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:11,123 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:11,123 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,125 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,125 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:11,125 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:11,125 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:11,125 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,125 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:11,126 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:11,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:11,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:11,128 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:11,128 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:11,128 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:11,128 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:11,128 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:11,129 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,129 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:11,129 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:11,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,130 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,131 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:11,132 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:11,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:11,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:11,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:11,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:11,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:11,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:11,133 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,133 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,133 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:11,133 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:11,133 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:11,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:11,201 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:58:11,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:11,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:11,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:11,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:11,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:11,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:11,233 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:11,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:11,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679893091219, max timestamp=1679893091232Num uniques 859; 2023-03-27 04:58:11,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-27 04:58:11,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679893091219, max timestamp=1679893091232Num uniques 859; 2023-03-27 04:58:11,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-27 04:58:11,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679893091219, max timestamp=1679893091232Num uniques 859; 2023-03-27 04:58:11,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-27 04:58:11,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:11,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:11,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:11,236 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:11,236 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:11,236 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:11,236 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:11,236 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,236 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,237 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,237 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,237 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:11,246 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:11,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:11,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:11,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:11,247 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:11,247 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:11,247 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:11,247 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:11,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:11,248 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:11,248 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:11,248 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:11,248 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:11,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:11,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:11,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:11,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:11,251 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:11,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eec5db[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:11,252 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:11,253 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,253 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:11,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:11,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:11,254 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,254 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:11,254 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:11,254 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:11,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,254 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:11,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:11,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:11,257 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:11,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:11,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:11,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:11,257 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:11,258 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,258 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:11,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,258 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,258 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,259 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,261 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:11,261 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:11,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:11,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:11,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:11,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:11,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:11,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:11,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,262 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,262 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:11,262 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:11,262 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:11,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:11,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893151254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43102 deadline: 1679893151255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893151266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893151267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893151268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,362 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:11,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:11,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:11,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:11,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:11,365 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:11,365 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:11,365 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:11,365 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:11,366 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,366 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,366 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:11,366 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:11,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:11,366 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,366 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:11,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:11,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:11,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:11,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:11,367 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:11,367 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:11,367 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:11,367 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:11,367 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:11,367 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:11,368 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:11,368 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:11,368 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:11,368 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:11,368 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:11,368 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,368 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:11,368 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:11,369 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,369 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,369 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,369 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893151368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893151368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:11,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61652ab1[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:11,376 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:11,376 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893151375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893151375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893151375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,384 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,384 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:11,384 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:11,384 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,384 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:11,384 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:11,385 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,385 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:11,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,385 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:11,385 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:11,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:11,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:11,387 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:11,387 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:11,387 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:11,387 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:11,387 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:11,388 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,388 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,388 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:11,389 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:11,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,390 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,392 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:11,392 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:11,392 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,392 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,392 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:11,392 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:11,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:11,392 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:11,392 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,392 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:11,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:11,392 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,393 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:11,393 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:11,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:11,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:11,493 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:11,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:11,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:11,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:11,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:11,495 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:11,495 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:11,495 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:11,496 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:11,496 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,496 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,497 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:11,497 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:11,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:11,497 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,497 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:11,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:11,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:11,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:11,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:11,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:11,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:11,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:11,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:11,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:11,498 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:11,498 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:11,498 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:11,498 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:11,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:11,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:11,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:11,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:11,499 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,499 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,499 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:11,500 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:11,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@186d20d6[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:11,502 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:11,502 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,502 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,502 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:11,503 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:11,503 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:11,503 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,503 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:11,504 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,504 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:11,504 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,504 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:11,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:11,506 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:11,506 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:11,506 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:11,506 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:11,506 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:11,508 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,509 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,509 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:11,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:11,509 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,510 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:11,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:11,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:11,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:11,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:11,512 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:11,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:11,512 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:11,512 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:11,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:11,512 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:11,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:11,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893151571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893151571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893151580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893151580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893151581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,610 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/20d92b6a753741a3a05da1e2f3a9f43a 2023-03-27 04:58:11,612 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:11,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:11,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:11,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:11,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:11,616 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:11,616 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:11,616 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:11,616 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:11,617 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,617 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,617 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,617 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:11,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:11,618 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,618 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:11,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:11,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:11,619 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:11,619 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:11,619 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:11,619 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:11,619 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:11,619 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:11,619 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:11,619 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:11,619 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:11,619 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:11,619 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:11,619 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,619 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:11,619 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:11,620 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,620 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,621 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,621 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:11,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:11,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17c32531[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:11,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:11,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,622 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,622 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:11,622 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:11,622 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:11,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:11,623 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,623 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,623 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:11,623 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:11,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:11,625 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:11,625 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:11,625 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:11,625 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:11,625 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:11,626 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,626 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,626 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:11,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:11,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:11,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,627 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,628 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:11,629 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:11,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:11,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:11,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:11,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:11,630 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,630 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,630 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,630 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,630 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:11,630 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:11,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:11,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:11,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:11,730 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:11,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:11,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:11,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:11,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:11,733 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:11,733 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:11,733 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:11,733 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:11,733 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,733 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,734 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:11,734 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:11,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:11,735 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,735 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:11,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:11,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:11,739 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:11,739 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:11,739 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:11,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:11,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:11,740 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:11,740 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:11,740 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:11,740 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:11,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:11,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:11,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:11,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:11,741 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:11,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63f7fa0f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:11,742 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:11,742 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,743 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,743 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:11,743 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:11,743 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:11,743 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,743 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:11,743 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:11,743 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,744 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:11,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,744 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:11,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:11,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:11,745 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:11,745 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:11,745 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:11,745 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:11,745 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:11,746 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,746 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,746 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:11,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:11,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,748 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,748 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:11,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:11,750 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:11,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:11,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:11,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:11,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:11,750 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:11,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:11,750 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:11,750 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:11,850 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:11,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:11,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:11,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:11,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:11,853 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:11,853 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:11,853 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:11,853 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:11,853 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,853 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,854 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:11,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:11,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:11,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:11,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:11,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:11,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:11,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:11,855 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:11,855 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:11,855 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:11,855 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:11,855 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:11,855 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:11,856 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:11,856 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:11,856 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:11,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:11,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:11,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:11,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:11,856 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:11,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b0f7462[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:11,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:11,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,858 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,858 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:11,858 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:11,859 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,859 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:11,859 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:11,859 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,859 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:11,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,859 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:11,859 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:11,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:11,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:11,861 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:11,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:11,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:11,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:11,861 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:11,862 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,862 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,862 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:11,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:11,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:11,865 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:11,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:11,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:11,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:11,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:11,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:11,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:11,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,866 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,866 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,866 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:11,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:11,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:11,866 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:11,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893151874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893151875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893151886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893151886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:11,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893151887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,966 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:11,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:11,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:11,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:11,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:11,968 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:11,968 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:11,968 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:11,968 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:11,969 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,969 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,969 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:11,970 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:11,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:11,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:11,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:11,971 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,971 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:11,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:11,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:11,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:11,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:11,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:11,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:11,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:11,971 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:11,971 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:11,971 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:11,971 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:11,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:11,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:11,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:11,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:11,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,972 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:11,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a158a45[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:11,973 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:11,973 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,974 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,974 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:11,974 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:11,974 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,974 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:11,974 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:11,975 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:11,975 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:11,975 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,975 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:11,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:11,976 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:11,976 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:11,976 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:11,976 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:11,977 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:11,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:11,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:11,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:11,979 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:11,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:11,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:11,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:11,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:11,981 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:11,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:11,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:11,981 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:11,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:11,981 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:11,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:11,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:11,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:11,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:11,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:12,019 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/ebfc1d35233f447b8481b5c36117c19a 2023-03-27 04:58:12,081 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:12,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:12,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:12,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:12,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:12,084 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:12,084 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:12,084 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:12,084 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:12,084 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,084 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,085 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,085 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:12,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:12,085 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,085 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:12,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:12,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:12,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:12,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:12,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:12,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:12,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:12,087 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,087 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:12,087 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:12,087 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,087 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:12,087 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:12,087 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,087 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:12,087 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:12,088 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,088 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,088 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,088 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:12,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54f1c69b[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:12,089 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:12,089 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,090 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:12,090 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,090 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:12,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:12,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:12,090 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,090 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:12,091 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,091 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:12,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:12,093 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:12,093 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:12,093 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:12,093 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:12,093 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:12,094 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,094 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,094 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:12,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,094 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,095 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,096 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:12,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:12,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:12,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:12,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:12,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:12,096 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:12,097 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:12,097 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:12,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:12,197 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:12,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:12,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:12,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:12,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:12,199 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:12,200 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:12,200 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:12,200 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:12,200 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,200 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,201 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:12,201 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:12,201 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,201 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:12,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:12,202 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:12,202 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:12,202 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:12,202 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:12,202 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:12,202 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,202 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:12,202 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:12,202 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,202 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:12,202 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:12,202 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,202 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:12,202 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:12,203 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,203 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,203 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,203 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:12,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@88a6988[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:12,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:12,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,204 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,205 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:12,205 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,205 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:12,205 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:12,205 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:12,205 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,205 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,205 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:12,205 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:12,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:12,207 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:12,207 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:12,207 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:12,207 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:12,207 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:12,208 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,208 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,208 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:12,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:12,208 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,209 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,212 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,212 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,212 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,212 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,212 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:12,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:12,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:12,212 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:12,212 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:12,213 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:12,213 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:12,213 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:12,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:12,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:12,213 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,213 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:12,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:12,313 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:12,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:12,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:12,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:12,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:12,318 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:12,318 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:12,318 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:12,318 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:12,318 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,318 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,319 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:12,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:12,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:12,319 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,320 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,320 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:12,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:12,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:12,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:12,321 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:12,321 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:12,321 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:12,321 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:12,321 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:12,321 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,321 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:12,321 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:12,321 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,321 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:12,321 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:12,321 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,321 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:12,321 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:12,322 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,322 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,322 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,322 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:12,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5876a09f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:12,323 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:12,323 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,324 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,324 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:12,324 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:12,324 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,324 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:12,324 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:12,324 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:12,324 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:12,324 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,324 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:12,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:12,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:12,326 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:12,326 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:12,326 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:12,326 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:12,326 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:12,327 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,327 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,327 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:12,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:12,328 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,329 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,330 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,330 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,330 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,330 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:12,330 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:12,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:12,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:12,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:12,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:12,330 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:12,330 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,330 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,330 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,330 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:12,330 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,330 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:12,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:12,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:12,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:12,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893152380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:12,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893152383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:12,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893152392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:12,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893152394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:12,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893152394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,428 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/00320c6322954030a84c6618d8b5c341 2023-03-27 04:58:12,431 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:12,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:12,431 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/20d92b6a753741a3a05da1e2f3a9f43a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/20d92b6a753741a3a05da1e2f3a9f43a 2023-03-27 04:58:12,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:12,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:12,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:12,433 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:12,433 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:12,433 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:12,433 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:12,434 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,434 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,434 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,434 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:12,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:12,434 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,435 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:12,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:12,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:12,435 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/20d92b6a753741a3a05da1e2f3a9f43a, entries=100, sequenceid=245, filesize=9.4 K 2023-03-27 04:58:12,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:12,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:12,435 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:12,435 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:12,435 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:12,435 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:12,436 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:12,436 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,436 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:12,436 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:12,436 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,436 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/ebfc1d35233f447b8481b5c36117c19a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/ebfc1d35233f447b8481b5c36117c19a 2023-03-27 04:58:12,436 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:12,436 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:12,436 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,436 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:12,436 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:12,437 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,437 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,437 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,437 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:12,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ec74e76[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:12,439 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:12,439 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,440 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,440 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:12,440 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:12,440 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:12,440 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/ebfc1d35233f447b8481b5c36117c19a, entries=100, sequenceid=245, filesize=9.4 K 2023-03-27 04:58:12,441 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,441 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:12,441 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/00320c6322954030a84c6618d8b5c341 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/00320c6322954030a84c6618d8b5c341 2023-03-27 04:58:12,441 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,441 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,441 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:12,441 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:12,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:12,443 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:12,443 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:12,443 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:12,443 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:12,443 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:12,443 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/00320c6322954030a84c6618d8b5c341, entries=100, sequenceid=245, filesize=9.4 K 2023-03-27 04:58:12,444 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,444 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,444 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:12,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,444 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 72e4f7cba88d8e5eeeccc51fc07b799f in 1243ms, sequenceid=245, compaction requested=true 2023-03-27 04:58:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,444 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:12,444 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,444 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:12,444 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:12,444 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:12,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:12,444 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,445 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34192 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,445 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/A is initiating minor compaction (all files) 2023-03-27 04:58:12,445 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34192 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:12,445 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/A in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,445 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/B is initiating minor compaction (all files) 2023-03-27 04:58:12,445 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/87c00c1c6e564e7ca5a6867820cf5f08, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/695b1353189a4b039c2f6d0579d03251, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/20d92b6a753741a3a05da1e2f3a9f43a] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=33.4 K 2023-03-27 04:58:12,445 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/B in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,446 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/bab189c8c3264e71be3ea9cc51afdfb6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/8e2c756acdcc4e3d93bfbeeef1eff684, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/ebfc1d35233f447b8481b5c36117c19a] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=33.4 K 2023-03-27 04:58:12,446 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 87c00c1c6e564e7ca5a6867820cf5f08, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679893089026 2023-03-27 04:58:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,446 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting bab189c8c3264e71be3ea9cc51afdfb6, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679893089026 2023-03-27 04:58:12,446 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 695b1353189a4b039c2f6d0579d03251, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1679893089082 2023-03-27 04:58:12,446 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,446 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 8e2c756acdcc4e3d93bfbeeef1eff684, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1679893089082 2023-03-27 04:58:12,446 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 20d92b6a753741a3a05da1e2f3a9f43a, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679893091199 2023-03-27 04:58:12,446 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting ebfc1d35233f447b8481b5c36117c19a, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679893091199 2023-03-27 04:58:12,447 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,449 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:12,449 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:12,449 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:12,449 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:12,449 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,450 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,450 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,450 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,450 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,450 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:12,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:12,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:12,450 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:12,450 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:12,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:12,453 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#A#compaction#415 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:12,453 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#B#compaction#416 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:12,550 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:12,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:12,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:12,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:12,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:12,553 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:12,553 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:12,553 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:12,553 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:12,553 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,553 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,554 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,554 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:12,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:12,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:12,554 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,554 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:12,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:12,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:12,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:12,555 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:12,555 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:12,555 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:12,555 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:12,555 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:12,555 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,555 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:12,556 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-27 04:58:12,556 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:12,556 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:12,556 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:12,556 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:12,556 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:12,556 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:12,569 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/ec1cfcce4a2443669239d674c144fcd8 2023-03-27 04:58:12,583 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/17183f4335694e079bcd96563c387ef6 2023-03-27 04:58:12,591 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/3f006ac3af3e4739ae3f4eb25460e0ba 2023-03-27 04:58:12,594 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/ec1cfcce4a2443669239d674c144fcd8 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/ec1cfcce4a2443669239d674c144fcd8 2023-03-27 04:58:12,597 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/ec1cfcce4a2443669239d674c144fcd8, entries=150, sequenceid=280, filesize=11.9 K 2023-03-27 04:58:12,597 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/17183f4335694e079bcd96563c387ef6 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/17183f4335694e079bcd96563c387ef6 2023-03-27 04:58:12,601 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/17183f4335694e079bcd96563c387ef6, entries=150, sequenceid=280, filesize=11.9 K 2023-03-27 04:58:12,602 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/3f006ac3af3e4739ae3f4eb25460e0ba as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/3f006ac3af3e4739ae3f4eb25460e0ba 2023-03-27 04:58:12,604 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/3f006ac3af3e4739ae3f4eb25460e0ba, entries=150, sequenceid=280, filesize=11.9 K 2023-03-27 04:58:12,605 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=0 B/0 for 72e4f7cba88d8e5eeeccc51fc07b799f in 50ms, sequenceid=280, compaction requested=true 2023-03-27 04:58:12,605 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:12,605 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,605 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:12,605 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:12,605 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,605 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:12,605 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:12,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,607 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,607 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,607 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:12,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:12,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74d8802b[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:12,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:12,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,610 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,610 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:12,610 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:12,610 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:12,610 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,610 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:12,610 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,610 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:12,611 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,611 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:12,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:12,614 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:12,614 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:12,614 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:12,614 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:12,614 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:12,615 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,615 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:12,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,615 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,616 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,617 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,618 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:12,618 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:12,618 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:12,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:12,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:12,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:12,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:12,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:12,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:12,618 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:12,618 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:12,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:12,619 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,619 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,619 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,619 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,619 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,619 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,619 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,718 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:12,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:12,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:12,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:12,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:12,722 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:12,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:12,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:12,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:12,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,723 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:12,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:12,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:12,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:12,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:12,724 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:12,724 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:12,725 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:12,725 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:12,725 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:12,725 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,725 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:12,725 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:12,725 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,725 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:12,725 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:12,725 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,725 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:12,725 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:12,726 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,726 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,726 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,726 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:12,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f456111[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:12,727 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:12,727 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,727 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,728 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,728 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:12,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:12,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:12,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:12,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:12,728 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:12,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:12,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:12,730 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:12,730 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:12,730 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:12,730 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:12,730 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:12,731 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,731 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,731 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:12,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,732 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,733 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,734 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:12,734 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:12,734 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:12,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:12,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:12,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:12,734 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:12,734 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:12,734 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,734 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:12,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:12,734 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,734 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,734 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,734 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,734 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,834 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:12,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:12,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:12,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:12,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:12,837 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:12,837 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:12,837 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:12,837 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:12,837 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,837 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,838 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:12,838 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:12,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:12,838 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,838 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:12,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:12,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:12,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:12,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:12,839 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:12,839 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:12,839 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:12,839 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:12,839 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:12,839 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,839 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:12,839 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:12,839 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,839 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:12,839 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:12,840 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,840 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:12,840 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:12,840 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,840 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,840 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,840 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:12,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:12,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42da4e3e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:12,842 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:12,842 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,843 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,843 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:12,843 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:12,843 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:12,843 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,843 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:12,843 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,843 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:12,843 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:12,844 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:12,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:12,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:12,845 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:12,845 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:12,845 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:12,845 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:12,845 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:12,846 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,846 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,846 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:12,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:12,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:12,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,847 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,848 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,849 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,849 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,849 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:12,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:12,849 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:12,849 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,849 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,849 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,849 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,850 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:12,850 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:12,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:12,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:12,850 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:12,850 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:12,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:12,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:12,867 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/ecebd15e605b48cc8527b4d1e29fb825 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/ecebd15e605b48cc8527b4d1e29fb825 2023-03-27 04:58:12,869 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/eba19d7f6d304698b7eccdfd931463c8 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/eba19d7f6d304698b7eccdfd931463c8 2023-03-27 04:58:12,870 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/A of 72e4f7cba88d8e5eeeccc51fc07b799f into ecebd15e605b48cc8527b4d1e29fb825(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:12,870 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:12,870 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/A, priority=13, startTime=1679893092444; duration=0sec 2023-03-27 04:58:12,870 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:12,870 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:58:12,871 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46378 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:58:12,872 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/C is initiating minor compaction (all files) 2023-03-27 04:58:12,872 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/C in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,872 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/f496c7da19864989ba87741d50697355, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/da4c427c71084eecbc33a1b70f8f9d5f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/00320c6322954030a84c6618d8b5c341, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/3f006ac3af3e4739ae3f4eb25460e0ba] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=45.3 K 2023-03-27 04:58:12,872 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting f496c7da19864989ba87741d50697355, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679893089026 2023-03-27 04:58:12,872 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting da4c427c71084eecbc33a1b70f8f9d5f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1679893089082 2023-03-27 04:58:12,873 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/B of 72e4f7cba88d8e5eeeccc51fc07b799f into eba19d7f6d304698b7eccdfd931463c8(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:12,873 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:12,873 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/B, priority=13, startTime=1679893092444; duration=0sec 2023-03-27 04:58:12,873 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:12,873 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 00320c6322954030a84c6618d8b5c341, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679893091199 2023-03-27 04:58:12,874 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 3f006ac3af3e4739ae3f4eb25460e0ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1679893091257 2023-03-27 04:58:12,879 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#C#compaction#420 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:12,892 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/a7ce4726b25e4f27aea578f8488e508f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/a7ce4726b25e4f27aea578f8488e508f 2023-03-27 04:58:12,896 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/C of 72e4f7cba88d8e5eeeccc51fc07b799f into a7ce4726b25e4f27aea578f8488e508f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:12,896 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:12,896 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/C, priority=12, startTime=1679893092444; duration=0sec 2023-03-27 04:58:12,896 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:12,950 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:12,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:12,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:12,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:12,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:12,954 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:12,954 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:12,954 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:12,954 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:12,954 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,954 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,955 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,955 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:12,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:12,955 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,955 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:12,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:12,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:12,957 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:12,957 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:12,957 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:12,957 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:12,957 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:12,957 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,957 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:12,957 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:12,957 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:12,957 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:12,957 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:12,957 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,957 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:12,957 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:12,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,958 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,959 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,960 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:12,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56fa1ced[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:12,961 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:12,961 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,961 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,961 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:12,962 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:12,962 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:12,962 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,962 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:12,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,962 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,962 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:12,962 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:12,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:12,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:12,964 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:12,964 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:12,964 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:12,964 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:12,964 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:12,965 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:12,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:12,965 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,965 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:12,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:12,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:12,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:12,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:12,966 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:12,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,968 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:12,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:12,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,969 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,969 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,969 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:12,969 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:12,969 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:12,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:12,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:12,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:12,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:12,969 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:12,969 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:12,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:12,969 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:12,969 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:12,969 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:12,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:12,970 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:12,970 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,070 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:13,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:13,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:13,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:13,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:13,072 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:13,073 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:13,073 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:13,073 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:13,073 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,073 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,074 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:13,074 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:13,074 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,074 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:13,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:13,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:13,078 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:13,078 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:13,078 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:13,079 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:13,079 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:13,079 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:13,079 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:13,079 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:13,079 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:13,079 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:13,079 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:13,079 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,079 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:13,079 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:13,084 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,084 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,085 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,085 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:13,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@669317c1[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:13,086 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:13,086 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,093 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,093 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:13,093 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:13,093 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:13,093 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,093 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:13,101 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:13,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:13,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:13,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:13,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:13,103 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:13,103 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:13,103 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:13,104 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:13,104 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:13,105 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,105 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:13,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,105 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:13,105 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,108 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:13,108 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:13,108 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:13,108 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:13,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:13,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:13,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:13,108 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,108 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,108 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:13,108 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,108 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:13,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:13,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:13,108 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,109 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:13,109 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,209 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:13,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:13,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:13,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:13,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:13,211 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:13,211 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:13,211 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:13,211 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:13,211 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,211 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,212 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,212 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:13,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:13,212 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,212 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:13,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:13,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:13,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:13,213 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:13,213 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:13,213 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:13,213 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:13,214 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:13,214 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:13,214 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:13,214 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:13,214 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:13,214 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:13,214 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:13,214 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,214 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:13,214 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:13,215 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,215 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,215 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,215 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:13,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62daafeb[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:13,216 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:13,216 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,217 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,217 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:13,217 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:13,217 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:13,217 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,217 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:13,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:13,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:13,218 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:13,218 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:13,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:13,220 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:13,220 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:13,220 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:13,220 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:13,220 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:13,221 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,221 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:13,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,221 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,222 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,223 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,224 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,224 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,224 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,224 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,224 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:13,224 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:13,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:13,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:13,224 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,224 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,224 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,224 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:13,224 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:13,224 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:13,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:13,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:13,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:13,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:13,325 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:13,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:13,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:13,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:13,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:13,327 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:13,327 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:13,327 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:13,327 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:13,328 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,328 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,328 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:13,328 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:13,328 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,329 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:13,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:13,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:13,329 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:13,329 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:13,329 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:13,330 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:13,330 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:13,330 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:13,330 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:13,330 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:13,330 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:13,330 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:13,330 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:13,330 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,330 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:13,330 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:13,331 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,331 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,331 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,331 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:13,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7261a630[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:13,334 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:13,334 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,334 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:13,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:13,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:13,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:13,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:13,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:13,336 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,336 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:13,336 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:13,336 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:13,336 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:13,336 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:13,336 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:13,336 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:13,337 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,337 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,337 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:13,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,338 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,339 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:13,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:13,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:13,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:13,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:13,340 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:13,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:13,341 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:13,341 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,341 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,341 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:13,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:13,402 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:58:13,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:13,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:13,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:13,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:13,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:13,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:13,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:13,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893153421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679893093410, max timestamp=1679893093426Num uniques 859; 2023-03-27 04:58:13,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-27 04:58:13,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679893093410, max timestamp=1679893093426Num uniques 859; 2023-03-27 04:58:13,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-27 04:58:13,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679893093410, max timestamp=1679893093426Num uniques 859; 2023-03-27 04:58:13,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-27 04:58:13,441 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:13,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:13,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:13,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:13,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:13,443 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:13,443 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:13,443 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:13,443 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:13,444 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,444 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,444 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:13,444 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:13,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:13,445 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,445 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:13,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:13,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:13,445 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:13,445 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:13,445 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:13,446 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:13,446 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:13,446 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:13,446 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:13,446 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:13,446 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:13,446 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:13,446 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:13,446 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,446 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:13,446 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:13,447 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,447 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,447 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,447 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:13,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74fad1e5[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:13,448 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:13,448 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,448 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,448 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:13,448 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:13,448 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,448 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:13,448 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:13,449 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:13,450 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:13,449 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,450 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:13,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:13,452 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:13,452 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:13,452 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:13,452 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:13,452 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:13,452 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,452 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:13,452 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:13,455 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,456 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:13,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893153446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,457 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,457 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:13,457 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,457 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:13,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:13,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:13,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:13,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:13,457 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:13,457 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,457 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,458 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,458 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:13,458 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:13,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:13,458 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:13,458 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:13,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893153450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:13,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893153451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:13,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893153452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:13,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893153529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,558 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:13,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:13,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:13,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893153557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:13,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:13,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:13,560 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:13,560 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:13,560 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:13,560 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:13,561 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,561 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,561 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:13,561 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:13,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:13,562 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,562 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:13,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:13,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:13,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:13,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:13,562 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:13,562 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:13,562 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:13,563 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:13,563 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:13,563 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:13,563 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:13,563 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:13,563 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:13,563 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:13,563 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:13,563 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,563 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:13,563 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:13,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:13,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893153559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:13,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893153559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,564 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,564 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,564 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,564 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:13,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aa9eda4[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:13,565 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:13,565 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,566 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,566 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:13,566 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,566 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:13,566 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:13,566 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:13,566 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,566 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:13,566 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:13,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:13,567 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:13,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893153562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:13,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:13,568 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:13,568 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:13,568 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:13,568 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:13,568 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:13,569 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,569 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,569 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:13,569 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,570 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,572 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,572 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,573 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:13,573 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:13,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:13,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:13,573 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:13,573 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:13,573 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,573 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:13,573 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:13,573 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:13,573 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:13,573 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,673 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:13,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:13,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:13,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:13,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:13,675 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:13,675 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:13,676 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:13,676 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:13,676 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,676 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,676 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,676 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:13,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:13,677 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,677 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:13,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:13,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:13,678 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:13,678 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:13,678 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:13,678 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:13,678 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:13,678 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:13,678 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:13,678 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:13,678 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:13,678 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:13,678 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:13,678 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,678 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:13,678 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:13,679 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,679 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,679 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,679 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:13,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d3bbeed[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:13,680 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:13,680 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,681 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,681 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:13,681 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:13,681 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:13,681 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,681 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:13,681 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,681 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:13,682 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,682 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:13,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:13,683 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:13,683 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:13,683 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:13,683 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:13,683 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:13,685 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,685 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,685 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:13,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:13,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:13,685 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,687 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,688 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:13,688 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,688 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:13,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:13,688 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:13,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:13,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:13,688 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,689 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:13,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:13,689 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,689 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:13,689 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:13,689 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,689 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,689 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:13,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893153733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:13,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893153761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:13,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893153764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:13,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893153765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:13,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893153768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,789 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:13,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:13,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:13,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:13,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:13,792 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:13,792 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:13,792 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:13,792 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:13,793 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,793 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,793 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,793 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:13,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:13,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:13,794 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,794 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:13,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:13,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:13,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:13,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:13,794 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:13,794 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:13,795 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:13,795 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:13,795 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:13,795 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:13,795 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:13,795 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:13,795 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:13,795 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:13,795 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:13,795 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,795 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:13,795 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:13,796 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,796 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,796 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,796 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:13,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4294cf5c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:13,797 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:13,797 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,798 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,798 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:13,798 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:13,798 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:13,798 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,798 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:13,799 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,799 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:13,799 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:13,799 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:13,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:13,800 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:13,800 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:13,800 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:13,800 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:13,800 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:13,802 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,802 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:13,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,802 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:13,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:13,802 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,803 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,804 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,804 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,804 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:13,804 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:13,804 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:13,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:13,804 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:13,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:13,804 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,804 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,804 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:13,804 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:13,805 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:13,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:13,805 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:13,813 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/9b51dc88efd24f0590f05b180c25eb39 2023-03-27 04:58:13,820 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/00e3ff56049141eaa826ca03f749d3e9 2023-03-27 04:58:13,826 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/9b55aa9fb33a4196acc130c407e2994a 2023-03-27 04:58:13,829 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/9b51dc88efd24f0590f05b180c25eb39 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/9b51dc88efd24f0590f05b180c25eb39 2023-03-27 04:58:13,832 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/9b51dc88efd24f0590f05b180c25eb39, entries=250, sequenceid=300, filesize=16.7 K 2023-03-27 04:58:13,832 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/00e3ff56049141eaa826ca03f749d3e9 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/00e3ff56049141eaa826ca03f749d3e9 2023-03-27 04:58:13,835 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/00e3ff56049141eaa826ca03f749d3e9, entries=150, sequenceid=300, filesize=11.9 K 2023-03-27 04:58:13,835 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/9b55aa9fb33a4196acc130c407e2994a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/9b55aa9fb33a4196acc130c407e2994a 2023-03-27 04:58:13,838 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/9b55aa9fb33a4196acc130c407e2994a, entries=150, sequenceid=300, filesize=11.9 K 2023-03-27 04:58:13,838 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 72e4f7cba88d8e5eeeccc51fc07b799f in 436ms, sequenceid=300, compaction requested=true 2023-03-27 04:58:13,838 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:13,839 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:13,839 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:13,839 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:13,839 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:13,839 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:13,840 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41868 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:13,840 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:13,840 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/A is initiating minor compaction (all files) 2023-03-27 04:58:13,840 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/B is initiating minor compaction (all files) 2023-03-27 04:58:13,840 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/A in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:13,840 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/B in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:13,840 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/ecebd15e605b48cc8527b4d1e29fb825, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/ec1cfcce4a2443669239d674c144fcd8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/9b51dc88efd24f0590f05b180c25eb39] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=40.9 K 2023-03-27 04:58:13,840 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/eba19d7f6d304698b7eccdfd931463c8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/17183f4335694e079bcd96563c387ef6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/00e3ff56049141eaa826ca03f749d3e9] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=36.1 K 2023-03-27 04:58:13,841 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting ecebd15e605b48cc8527b4d1e29fb825, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679893089087 2023-03-27 04:58:13,841 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting eba19d7f6d304698b7eccdfd931463c8, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679893089087 2023-03-27 04:58:13,841 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting ec1cfcce4a2443669239d674c144fcd8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1679893091257 2023-03-27 04:58:13,841 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 17183f4335694e079bcd96563c387ef6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1679893091257 2023-03-27 04:58:13,841 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 9b51dc88efd24f0590f05b180c25eb39, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1679893093395 2023-03-27 04:58:13,841 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 00e3ff56049141eaa826ca03f749d3e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1679893093395 2023-03-27 04:58:13,846 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#A#compaction#425 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:13,847 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#B#compaction#424 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:13,905 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:13,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:13,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:13,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:13,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:13,908 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:13,908 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:13,908 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:13,908 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:13,908 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,908 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,909 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:13,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:13,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:13,909 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,909 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:13,909 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:13,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:13,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:13,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:13,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:13,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:13,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:13,910 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:13,910 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:13,910 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:13,910 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:13,910 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:13,910 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:13,910 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:13,910 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-03-27 04:58:13,910 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:13,910 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:13,910 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:13,911 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:13,911 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:13,911 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:13,918 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/b54a49a8d040422bbab0feab9b224ffe 2023-03-27 04:58:13,925 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/3bcc7e036c574c2aa98282d6b97b700c 2023-03-27 04:58:14,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:14,044 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:14,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893154074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893154096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893154096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893154096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893154096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893154196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893154203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893154203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893154203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893154203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,254 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/bc265aeee051489784054abb58a78ca1 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/bc265aeee051489784054abb58a78ca1 2023-03-27 04:58:14,254 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/fcb05b2041404e3eb30b5654cfcd0b03 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/fcb05b2041404e3eb30b5654cfcd0b03 2023-03-27 04:58:14,257 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/A of 72e4f7cba88d8e5eeeccc51fc07b799f into bc265aeee051489784054abb58a78ca1(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:14,257 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/B of 72e4f7cba88d8e5eeeccc51fc07b799f into fcb05b2041404e3eb30b5654cfcd0b03(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:14,257 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:14,257 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:14,257 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/A, priority=13, startTime=1679893093839; duration=0sec 2023-03-27 04:58:14,257 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/B, priority=13, startTime=1679893093839; duration=0sec 2023-03-27 04:58:14,257 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:14,257 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:14,257 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-27 04:58:14,258 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:58:14,258 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:58:14,258 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. because compaction request was cancelled 2023-03-27 04:58:14,335 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/2c381f23393b4a7f984d70e593f50fdb 2023-03-27 04:58:14,338 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/b54a49a8d040422bbab0feab9b224ffe as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/b54a49a8d040422bbab0feab9b224ffe 2023-03-27 04:58:14,341 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/b54a49a8d040422bbab0feab9b224ffe, entries=150, sequenceid=333, filesize=11.9 K 2023-03-27 04:58:14,342 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/3bcc7e036c574c2aa98282d6b97b700c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/3bcc7e036c574c2aa98282d6b97b700c 2023-03-27 04:58:14,345 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/3bcc7e036c574c2aa98282d6b97b700c, entries=150, sequenceid=333, filesize=11.9 K 2023-03-27 04:58:14,345 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/2c381f23393b4a7f984d70e593f50fdb as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/2c381f23393b4a7f984d70e593f50fdb 2023-03-27 04:58:14,348 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/2c381f23393b4a7f984d70e593f50fdb, entries=150, sequenceid=333, filesize=11.9 K 2023-03-27 04:58:14,348 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=67.09 KB/68700 for 72e4f7cba88d8e5eeeccc51fc07b799f in 438ms, sequenceid=333, compaction requested=true 2023-03-27 04:58:14,348 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:14,348 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:14,349 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:14,349 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:14,349 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,349 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:14,349 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:14,350 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,350 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,350 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,350 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:14,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60435b12[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:14,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:14,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,351 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,351 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:14,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:14,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:14,352 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,352 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:14,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:14,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:14,352 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,352 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:14,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:14,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:14,354 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:14,354 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:14,355 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:14,355 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:14,355 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:14,355 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,355 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,355 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:14,356 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,356 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:14,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:14,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:14,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:14,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,358 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:14,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:14,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:14,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:14,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:14,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,358 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,358 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:14,358 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:14,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:14,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:14,402 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-27 04:58:14,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:14,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:14,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:14,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:14,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:14,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:14,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893154422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893154422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893154423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893154423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893154424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,459 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:14,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:14,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:14,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:14,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:14,461 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:14,461 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:14,461 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:14,461 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:14,461 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,461 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,462 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,462 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:14,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:14,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:14,462 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,462 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:14,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:14,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:14,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:14,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:14,463 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:14,463 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:14,463 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:14,463 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:14,463 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:14,463 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:14,463 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:14,463 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:14,463 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:14,463 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:14,463 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:14,463 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,463 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:14,463 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:14,464 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,464 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,464 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,464 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:14,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25c25e73[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:14,465 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:14,465 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,466 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,466 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:14,466 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:14,466 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:14,466 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,466 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:14,466 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,466 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:14,466 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,467 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:14,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:14,468 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:14,468 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:14,468 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:14,468 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:14,468 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:14,469 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,469 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,469 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,469 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,470 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,472 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:14,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:14,472 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:14,472 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,472 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:14,472 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:14,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:14,472 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:14,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:14,472 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,472 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,472 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,472 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,472 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:14,472 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:14,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:14,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:14,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893154528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893154528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893154533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893154533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893154534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,573 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:14,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:14,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:14,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:14,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:14,575 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:14,575 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:14,575 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:14,575 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:14,576 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,576 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,576 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,576 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:14,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:14,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:14,576 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,576 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:14,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:14,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:14,577 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:14,577 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:14,577 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:14,577 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:14,577 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:14,577 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:14,577 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:14,578 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:14,578 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:14,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:14,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:14,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:14,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:14,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,578 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,579 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,579 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:14,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ee40f0[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:14,580 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:14,580 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,580 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,580 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:14,580 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:14,580 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,581 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:14,581 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:14,581 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,581 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:14,581 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:14,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,581 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:14,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:14,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:14,584 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:14,584 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:14,584 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:14,584 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:14,584 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:14,585 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,585 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,585 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:14,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:14,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,587 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,588 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,589 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,589 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:14,589 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,590 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:14,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:14,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:14,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:14,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:14,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:14,590 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:14,590 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,590 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:14,590 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,590 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:14,590 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,590 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,590 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:14,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:14,690 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:14,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:14,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:14,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:14,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:14,693 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:14,693 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:14,693 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:14,693 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:14,693 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,693 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,694 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,694 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:14,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:14,694 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,694 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:14,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:14,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:14,695 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:14,695 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:14,695 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:14,695 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:14,695 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:14,695 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:14,695 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:14,695 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:14,695 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:14,695 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:14,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:14,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:14,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:14,696 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,697 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,697 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:14,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2164993e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:14,698 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:14,698 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,698 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,699 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:14,699 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:14,699 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,699 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:14,699 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:14,699 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,699 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,699 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:14,700 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:14,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:14,701 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:14,701 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:14,701 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:14,701 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:14,701 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:14,702 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,702 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,702 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:14,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:14,702 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,703 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,705 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,705 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,705 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,705 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,705 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,705 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:14,705 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:14,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:14,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:14,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:14,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:14,705 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:14,705 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:14,705 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,705 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:14,705 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:14,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893154731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893154731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893154740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893154741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:14,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893154741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,805 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:14,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:14,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:14,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:14,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:14,808 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:14,808 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:14,808 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:14,808 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:14,808 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,808 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,809 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,809 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:14,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:14,809 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,809 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:14,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:14,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:14,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:14,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:14,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:14,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:14,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:14,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:14,810 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:14,810 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:14,810 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:14,810 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:14,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:14,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:14,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:14,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:14,811 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,811 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,811 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,811 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:14,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b85ddcb[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:14,812 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:14,812 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,813 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,813 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:14,813 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:14,813 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:14,813 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,813 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:14,814 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,814 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:14,814 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,814 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:14,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:14,815 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:14,816 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:14,816 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:14,816 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:14,816 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:14,816 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,816 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,816 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:14,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,817 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,818 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,819 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,819 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,819 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,819 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:14,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:14,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:14,819 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:14,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:14,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:14,819 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:14,819 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,819 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,819 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,819 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,819 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:14,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:14,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:14,819 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:14,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:14,824 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/d0bff69d73324c36ab9d72520f6e902f 2023-03-27 04:58:14,920 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:14,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:14,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:14,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:14,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:14,922 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:14,922 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:14,922 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:14,922 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:14,922 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,923 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,923 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:14,923 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:14,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:14,923 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,923 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:14,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:14,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:14,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:14,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:14,924 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:14,924 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:14,924 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:14,924 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:14,924 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:14,924 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:14,925 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:14,925 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:14,925 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:14,925 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:14,925 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:14,925 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,925 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:14,925 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:14,926 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,926 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,926 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,926 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:14,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:14,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c83310e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:14,927 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:14,927 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,927 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,928 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:14,928 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:14,928 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:14,928 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,928 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:14,929 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,929 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,929 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:14,929 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:14,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:14,931 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:14,931 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:14,931 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:14,931 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:14,931 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:14,932 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,932 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:14,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:14,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,932 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:14,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:14,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:14,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:14,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,933 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:14,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:14,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,934 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:14,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:14,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,936 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:14,936 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:14,936 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:14,936 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:14,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:14,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:14,936 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,936 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:14,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:14,936 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:14,936 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:14,936 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,936 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:14,936 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:14,936 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,036 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:15,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:15,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:15,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:15,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:15,039 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:15,039 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:15,039 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:15,039 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:15,039 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,039 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,040 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:15,040 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:15,040 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,040 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:15,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:15,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:15,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:15,041 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:15,041 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:15,041 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:15,041 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:15,041 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:15,041 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:15,041 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:15,041 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:15,041 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:15,041 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:15,041 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:15,041 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,041 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:15,041 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:15,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:15,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893155038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:15,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893155038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,042 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,042 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,042 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,042 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:15,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:15,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bdd05ae[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:15,043 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:15,043 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,044 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,044 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:15,044 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:15,044 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:15,044 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,044 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:15,045 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:15,045 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:15,045 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,045 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:15,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:15,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:15,047 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:15,047 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:15,047 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:15,047 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:15,047 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:15,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:15,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893155045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:15,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893155045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:15,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893155045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,048 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,048 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:15,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,048 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:15,049 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,050 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,051 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,051 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,051 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,051 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,051 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:15,051 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,051 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:15,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:15,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:15,051 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:15,051 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,051 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:15,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:15,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:15,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:15,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:15,052 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:15,052 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:15,152 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:15,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:15,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:15,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:15,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:15,155 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:15,155 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:15,155 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:15,155 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:15,156 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,156 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,156 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,156 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:15,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:15,157 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,157 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:15,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:15,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:15,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:15,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:15,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:15,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:15,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:15,158 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:15,158 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:15,158 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:15,158 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:15,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:15,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:15,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:15,158 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:15,159 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,159 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,159 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,159 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:15,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ef5ca82[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:15,161 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:15,161 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,161 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,161 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:15,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:15,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:15,162 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,162 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:15,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:15,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:15,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:15,163 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:15,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:15,165 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:15,165 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:15,165 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:15,165 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:15,165 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:15,166 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,166 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,166 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:15,166 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,168 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,169 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,169 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,169 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,169 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:15,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:15,169 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:15,169 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:15,169 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,169 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:15,169 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:15,169 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,170 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,170 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:15,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:15,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:15,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:15,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:15,232 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/70f733bf8a254d52895ba353ffeade73 2023-03-27 04:58:15,270 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:15,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:15,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:15,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:15,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:15,272 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:15,272 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:15,273 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:15,273 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:15,273 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,273 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,274 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:15,274 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:15,274 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,274 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:15,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:15,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:15,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:15,275 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:15,275 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:15,275 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:15,275 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:15,275 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:15,275 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:15,275 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:15,275 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:15,275 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:15,275 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:15,275 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:15,276 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,276 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:15,276 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:15,276 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,276 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,277 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,277 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:15,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:15,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b6b3f12[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:15,279 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:15,279 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,280 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,281 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:15,281 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:15,281 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:15,281 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,281 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:15,281 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:15,281 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,281 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,281 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:15,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:15,283 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:15,283 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:15,283 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:15,283 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:15,284 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:15,284 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,284 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:15,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,285 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,285 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:15,286 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,287 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:15,287 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:15,288 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,288 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,288 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:15,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:15,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:15,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:15,288 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:15,288 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:15,288 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,288 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:15,288 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,288 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,288 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:15,388 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:15,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:15,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:15,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:15,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:15,391 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:15,391 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:15,391 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:15,391 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:15,391 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,391 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,392 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:15,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:15,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:15,392 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,392 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,392 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:15,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:15,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:15,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:15,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:15,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:15,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:15,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:15,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:15,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:15,393 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:15,393 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:15,393 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:15,393 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:15,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:15,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:15,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:15,393 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:15,395 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,395 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,395 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,395 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:15,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:15,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e957030[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:15,397 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:15,398 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,398 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,398 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:15,398 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:15,398 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:15,398 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,399 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:15,399 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:15,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:15,399 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:15,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:15,402 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:15,402 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:15,402 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:15,402 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:15,402 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:15,403 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,403 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,403 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,403 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,404 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,405 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:15,405 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,405 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:15,405 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:15,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:15,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:15,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:15,405 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,406 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,406 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,406 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:15,406 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:15,406 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:15,406 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,406 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:15,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:15,506 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:15,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:15,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:15,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:15,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:15,509 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:15,509 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:15,509 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:15,509 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:15,509 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,509 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,510 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,510 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:15,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:15,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:15,510 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,510 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:15,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:15,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,511 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:15,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:15,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:15,511 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:15,511 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:15,511 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:15,512 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:15,512 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:15,512 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:15,512 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:15,513 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:15,513 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:15,513 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:15,513 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:15,513 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,513 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:15,513 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:15,514 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,514 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,514 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,514 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:15,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:15,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dd6ee39[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:15,515 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:15,515 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,516 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,516 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:15,516 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:15,516 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,516 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:15,516 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:15,517 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,517 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:15,517 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:15,517 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:15,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:15,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:15,519 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:15,519 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:15,519 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:15,519 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:15,519 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:15,520 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,520 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:15,520 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:15,521 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,523 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,524 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:15,524 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:15,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:15,525 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:15,525 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:15,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:15,525 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,525 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,525 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,525 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,525 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,525 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,525 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:15,525 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:15,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:15,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:15,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:15,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893155543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:15,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893155544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:15,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893155549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:15,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893155550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:15,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893155552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,625 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:15,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:15,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:15,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:15,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:15,628 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:15,628 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:15,628 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:15,628 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:15,628 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,628 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,629 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:15,629 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:15,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:15,630 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,630 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:15,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:15,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:15,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:15,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:15,630 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:15,630 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:15,630 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:15,634 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:15,634 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:15,634 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:15,634 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:15,634 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:15,634 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:15,634 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:15,634 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:15,634 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,634 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:15,634 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:15,635 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,635 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:15,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:15,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d0798db[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:15,637 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:15,637 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,637 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,637 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:15,637 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:15,637 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:15,637 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,637 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:15,638 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,638 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:15,638 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:15,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,638 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:15,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:15,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:15,641 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:15,641 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:15,641 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:15,641 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:15,641 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:15,642 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,642 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:15,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:15,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,642 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:15,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:15,643 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:15,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,644 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:15,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:15,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:15,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:15,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:15,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:15,645 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:15,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:15,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:15,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:15,645 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:15,646 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:15,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:15,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:15,655 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/6af1b2fd78654e098de613732b9664aa 2023-03-27 04:58:15,658 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/d0bff69d73324c36ab9d72520f6e902f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/d0bff69d73324c36ab9d72520f6e902f 2023-03-27 04:58:15,661 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/d0bff69d73324c36ab9d72520f6e902f, entries=200, sequenceid=349, filesize=14.3 K 2023-03-27 04:58:15,662 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/70f733bf8a254d52895ba353ffeade73 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/70f733bf8a254d52895ba353ffeade73 2023-03-27 04:58:15,664 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/70f733bf8a254d52895ba353ffeade73, entries=150, sequenceid=349, filesize=11.9 K 2023-03-27 04:58:15,665 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/6af1b2fd78654e098de613732b9664aa as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/6af1b2fd78654e098de613732b9664aa 2023-03-27 04:58:15,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/6af1b2fd78654e098de613732b9664aa, entries=150, sequenceid=349, filesize=11.9 K 2023-03-27 04:58:15,668 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 72e4f7cba88d8e5eeeccc51fc07b799f in 1266ms, sequenceid=349, compaction requested=true 2023-03-27 04:58:15,668 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:15,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:15,668 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:15,668 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:15,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:15,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:15,669 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39680 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:15,669 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:15,669 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/A is initiating minor compaction (all files) 2023-03-27 04:58:15,669 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/B is initiating minor compaction (all files) 2023-03-27 04:58:15,669 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/A in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:15,669 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/B in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:15,669 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/bc265aeee051489784054abb58a78ca1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/b54a49a8d040422bbab0feab9b224ffe, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/d0bff69d73324c36ab9d72520f6e902f] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=38.8 K 2023-03-27 04:58:15,669 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/fcb05b2041404e3eb30b5654cfcd0b03, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/3bcc7e036c574c2aa98282d6b97b700c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/70f733bf8a254d52895ba353ffeade73] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=36.4 K 2023-03-27 04:58:15,669 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting bc265aeee051489784054abb58a78ca1, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1679893093395 2023-03-27 04:58:15,669 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting fcb05b2041404e3eb30b5654cfcd0b03, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1679893093395 2023-03-27 04:58:15,669 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting b54a49a8d040422bbab0feab9b224ffe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1679893093445 2023-03-27 04:58:15,669 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 3bcc7e036c574c2aa98282d6b97b700c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1679893093445 2023-03-27 04:58:15,670 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting d0bff69d73324c36ab9d72520f6e902f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1679893094075 2023-03-27 04:58:15,670 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 70f733bf8a254d52895ba353ffeade73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1679893094075 2023-03-27 04:58:15,674 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#B#compaction#432 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:15,676 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#A#compaction#433 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:15,746 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:15,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:15,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:15,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:15,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:15,748 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:15,748 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:15,748 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:15,748 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:15,749 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,749 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,749 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,749 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:15,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:15,750 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:15,750 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:15,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:15,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:15,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:15,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:15,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:15,750 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:15,750 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:15,750 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:15,751 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:15,751 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:15,751 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:15,751 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:15,751 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-27 04:58:15,751 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:15,751 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:15,751 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:15,751 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:15,751 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:15,751 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:16,089 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/361df267d0d64916b610993b1bf66e53 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/361df267d0d64916b610993b1bf66e53 2023-03-27 04:58:16,090 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/3524cc93c1fa492b8dfcca052f4e2bdf as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/3524cc93c1fa492b8dfcca052f4e2bdf 2023-03-27 04:58:16,093 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/A of 72e4f7cba88d8e5eeeccc51fc07b799f into 3524cc93c1fa492b8dfcca052f4e2bdf(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:16,093 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/B of 72e4f7cba88d8e5eeeccc51fc07b799f into 361df267d0d64916b610993b1bf66e53(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:16,093 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:16,093 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:16,093 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/A, priority=13, startTime=1679893095668; duration=0sec 2023-03-27 04:58:16,093 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/B, priority=13, startTime=1679893095668; duration=0sec 2023-03-27 04:58:16,093 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:16,093 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:16,093 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:58:16,094 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:58:16,094 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/C is initiating minor compaction (all files) 2023-03-27 04:58:16,094 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/C in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:16,094 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/a7ce4726b25e4f27aea578f8488e508f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/9b55aa9fb33a4196acc130c407e2994a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/2c381f23393b4a7f984d70e593f50fdb, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/6af1b2fd78654e098de613732b9664aa] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=48.2 K 2023-03-27 04:58:16,094 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting a7ce4726b25e4f27aea578f8488e508f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1679893091257 2023-03-27 04:58:16,095 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 9b55aa9fb33a4196acc130c407e2994a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1679893093395 2023-03-27 04:58:16,095 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 2c381f23393b4a7f984d70e593f50fdb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1679893093445 2023-03-27 04:58:16,095 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 6af1b2fd78654e098de613732b9664aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1679893094075 2023-03-27 04:58:16,101 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#C#compaction#435 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:16,108 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/99a6316591914eaa883b72c0c5762854 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/99a6316591914eaa883b72c0c5762854 2023-03-27 04:58:16,111 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/C of 72e4f7cba88d8e5eeeccc51fc07b799f into 99a6316591914eaa883b72c0c5762854(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:16,112 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:16,112 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/C, priority=12, startTime=1679893095668; duration=0sec 2023-03-27 04:58:16,112 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:16,170 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/8d1d90d342a7458283c01d8ff9144e6d 2023-03-27 04:58:16,179 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/c300c369d7f64ef9859cbcec3f71fae5 2023-03-27 04:58:16,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:16,555 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:16,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:16,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893156561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:16,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893156563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:16,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893156564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:16,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893156564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:16,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893156565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,585 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/d5a1950fdf2b436faca1b6da5eb16750 2023-03-27 04:58:16,589 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/8d1d90d342a7458283c01d8ff9144e6d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/8d1d90d342a7458283c01d8ff9144e6d 2023-03-27 04:58:16,592 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/8d1d90d342a7458283c01d8ff9144e6d, entries=150, sequenceid=367, filesize=11.9 K 2023-03-27 04:58:16,593 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/c300c369d7f64ef9859cbcec3f71fae5 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/c300c369d7f64ef9859cbcec3f71fae5 2023-03-27 04:58:16,596 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/c300c369d7f64ef9859cbcec3f71fae5, entries=150, sequenceid=367, filesize=11.9 K 2023-03-27 04:58:16,596 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/d5a1950fdf2b436faca1b6da5eb16750 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/d5a1950fdf2b436faca1b6da5eb16750 2023-03-27 04:58:16,600 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/d5a1950fdf2b436faca1b6da5eb16750, entries=150, sequenceid=367, filesize=11.9 K 2023-03-27 04:58:16,601 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 72e4f7cba88d8e5eeeccc51fc07b799f in 850ms, sequenceid=367, compaction requested=false 2023-03-27 04:58:16,601 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:16,601 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:16,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:16,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:16,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:16,601 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:16,602 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,602 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:16,603 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,603 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:16,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:16,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fe6a7c6[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:16,604 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:16,604 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,605 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,606 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:16,606 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,606 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:16,606 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:16,606 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:16,606 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:16,606 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:16,606 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:16,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:16,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:16,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:16,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:16,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:16,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:16,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:16,609 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:16,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:16,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:16,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:16,609 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:16,611 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:16,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:16,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:16,611 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:16,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,612 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:16,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,613 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,614 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:16,614 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:16,614 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,614 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,614 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,614 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:16,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:16,614 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:16,614 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,614 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:16,614 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:16,614 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:16,614 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:16,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:16,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:16,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:16,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:16,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:16,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:16,667 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-27 04:58:16,678 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:16,678 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:16,678 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:16,678 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:16,678 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:16,678 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:16,686 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/843ba54b49424efc8575d3917e3c3c50 2023-03-27 04:58:16,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:16,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:16,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893156688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893156689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:16,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:16,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893156695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893156695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:16,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893156695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,714 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:16,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:16,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:16,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:16,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:16,716 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:16,716 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:16,717 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:16,717 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:16,717 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,717 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:16,718 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:16,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:16,718 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:16,718 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,718 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:16,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:16,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:16,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:16,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:16,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:16,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:16,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:16,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:16,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:16,719 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:16,719 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:16,719 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:16,719 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:16,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:16,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:16,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:16,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:16,720 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:16,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:16,720 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,721 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,721 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:16,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:16,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a874efa[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:16,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:16,723 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,724 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:16,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:16,724 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:16,724 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:16,724 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:16,724 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:16,725 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:16,725 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:16,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:16,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:16,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:16,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:16,728 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:16,728 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:16,728 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:16,728 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:16,728 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:16,729 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,729 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:16,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,729 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:16,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,730 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,730 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:16,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,732 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,732 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:16,732 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:16,732 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:16,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:16,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:16,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:16,732 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:16,732 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:16,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:16,732 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:16,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:16,732 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,732 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:16,732 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,732 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,732 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:16,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:16,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:16,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893156796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:16,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893156796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:16,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893156801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:16,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893156801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:16,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893156801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,832 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:16,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:16,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:16,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:16,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:16,835 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:16,835 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:16,835 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:16,835 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:16,835 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,835 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:16,836 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,836 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:16,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:16,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:16,836 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,836 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:16,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:16,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:16,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:16,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:16,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:16,837 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:16,837 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:16,837 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:16,837 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:16,837 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:16,837 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:16,837 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:16,837 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:16,837 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:16,837 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:16,837 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:16,837 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,837 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:16,837 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:16,838 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,838 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:16,838 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,838 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:16,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:16,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40faccb5[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:16,839 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:16,839 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,840 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,840 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,840 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:16,840 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:16,840 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:16,840 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:16,841 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,841 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:16,841 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:16,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:16,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:16,841 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:16,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:16,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:16,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:16,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:16,842 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:16,842 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:16,842 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:16,842 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:16,842 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:16,843 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,843 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:16,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:16,843 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:16,844 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:16,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:16,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:16,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:16,845 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:16,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,846 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:16,846 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:16,846 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:16,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:16,846 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:16,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:16,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:16,846 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:16,846 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,846 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,846 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,846 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:16,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:16,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:16,846 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:16,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:16,846 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:16,846 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,947 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:16,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:16,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:16,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:16,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:16,949 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:16,949 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:16,949 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:16,949 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:16,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:16,950 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:16,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:16,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:16,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:16,951 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,951 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:16,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:16,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:16,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:16,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:16,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:16,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:16,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:16,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:16,951 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:16,951 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:16,951 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:16,951 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:16,952 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:16,952 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:16,952 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:16,952 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,952 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:16,952 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:16,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,952 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:16,953 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,953 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:16,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:16,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@611a5531[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:16,954 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:16,954 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,954 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,954 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:16,954 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:16,954 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:16,954 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,954 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:16,956 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,956 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:16,956 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:16,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:16,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:16,956 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:16,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:16,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:16,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:16,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:16,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:16,957 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:16,957 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:16,957 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:16,957 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:16,957 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:16,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:16,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,958 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:16,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:16,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,960 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:16,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:16,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:16,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:16,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:16,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:16,962 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:16,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:16,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:16,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:16,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:16,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:16,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:16,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,962 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:16,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:16,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:16,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:16,962 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:16,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893156998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893156999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893157004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893157004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893157005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,063 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:17,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:17,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:17,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:17,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:17,065 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:17,065 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:17,065 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:17,065 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:17,066 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,066 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,066 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:17,066 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:17,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:17,067 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,067 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:17,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:17,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:17,067 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:17,067 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:17,067 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:17,067 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:17,067 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:17,068 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:17,068 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:17,068 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:17,068 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:17,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:17,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:17,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:17,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:17,068 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,068 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:17,069 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,069 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:17,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:17,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b980643[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:17,070 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:17,070 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,070 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,071 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:17,071 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,071 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:17,071 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:17,071 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:17,071 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,071 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:17,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:17,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:17,071 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:17,072 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:17,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:17,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:17,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:17,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:17,073 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:17,073 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:17,073 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:17,073 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:17,073 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:17,074 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,074 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,074 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:17,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:17,074 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,075 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,077 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:17,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:17,077 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,077 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:17,077 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:17,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:17,077 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:17,077 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,077 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:17,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:17,077 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:17,077 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:17,077 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,077 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,093 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/2159dd68633c4fc0b9ae6549724eb475 2023-03-27 04:58:17,177 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:17,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:17,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:17,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:17,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:17,180 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:17,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:17,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:17,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:17,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,181 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:17,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:17,181 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:17,181 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,181 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:17,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:17,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:17,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:17,182 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:17,182 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:17,182 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:17,182 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:17,182 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:17,182 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:17,182 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:17,182 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:17,182 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:17,182 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:17,182 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:17,182 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,183 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:17,183 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:17,183 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,183 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:17,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:17,184 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,184 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:17,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:17,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:17,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:17,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:17,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b487f54[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:17,185 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:17,185 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,185 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,186 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:17,186 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:17,186 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:17,186 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,186 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:17,186 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,186 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:17,186 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:17,186 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:17,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:17,188 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:17,188 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:17,188 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:17,188 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:17,188 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:17,189 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,189 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:17,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,189 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:17,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:17,190 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,191 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,192 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,192 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,192 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,192 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:17,192 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:17,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:17,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:17,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:17,192 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:17,192 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:17,192 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,192 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:17,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:17,192 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,193 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,193 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:17,292 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:17,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:17,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:17,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:17,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:17,295 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:17,295 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:17,295 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:17,295 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:17,296 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,296 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,296 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:17,296 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:17,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:17,297 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,297 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:17,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:17,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:17,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:17,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:17,298 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:17,298 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:17,298 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:17,299 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:17,299 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:17,299 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:17,299 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:17,299 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:17,299 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:17,299 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:17,299 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:17,299 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,299 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:17,299 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:17,300 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,300 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:17,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:17,300 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,300 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:17,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:17,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:17,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:17,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:17,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dc197ec[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:17,301 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:17,301 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,302 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,302 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:17,302 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:17,302 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,302 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:17,302 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:17,302 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:17,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:17,302 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:17,303 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:17,303 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:17,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:17,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:17,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:17,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:17,304 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:17,304 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:17,304 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:17,304 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:17,304 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:17,305 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,305 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,305 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:17,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:17,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:17,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:17,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:17,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893157303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:17,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893157303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,307 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:17,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,308 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:17,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893157307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893157307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,309 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:17,309 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:17,310 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:17,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:17,310 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:17,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:17,310 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:17,310 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,310 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:17,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:17,310 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:17,310 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:17,310 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,310 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:17,310 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893157310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,410 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:17,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:17,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:17,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:17,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:17,413 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:17,413 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:17,413 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:17,413 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:17,413 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,413 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,414 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,414 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:17,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:17,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:17,414 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,414 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:17,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:17,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:17,415 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:17,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:17,415 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:17,415 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:17,415 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:17,415 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:17,415 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:17,415 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:17,415 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:17,415 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:17,415 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:17,415 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:17,415 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:17,415 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,415 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:17,415 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:17,416 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,416 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:17,416 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,416 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:17,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:17,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e41e8b8[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:17,417 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:17,418 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,418 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:17,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:17,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:17,418 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,418 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:17,419 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,419 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:17,419 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:17,419 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:17,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:17,421 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:17,421 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:17,421 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:17,421 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:17,421 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:17,422 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,422 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:17,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,422 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:17,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:17,422 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,423 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:17,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,424 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,424 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,424 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,424 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:17,424 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:17,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:17,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:17,424 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:17,424 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:17,425 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,425 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,425 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:17,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:17,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:17,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:17,425 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:17,425 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:17,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/81b7c0f3c44149dab28c84434d8e77ec 2023-03-27 04:58:17,504 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/843ba54b49424efc8575d3917e3c3c50 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/843ba54b49424efc8575d3917e3c3c50 2023-03-27 04:58:17,506 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/843ba54b49424efc8575d3917e3c3c50, entries=150, sequenceid=386, filesize=11.9 K 2023-03-27 04:58:17,507 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/2159dd68633c4fc0b9ae6549724eb475 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/2159dd68633c4fc0b9ae6549724eb475 2023-03-27 04:58:17,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/2159dd68633c4fc0b9ae6549724eb475, entries=150, sequenceid=386, filesize=11.9 K 2023-03-27 04:58:17,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/81b7c0f3c44149dab28c84434d8e77ec as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/81b7c0f3c44149dab28c84434d8e77ec 2023-03-27 04:58:17,519 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/81b7c0f3c44149dab28c84434d8e77ec, entries=150, sequenceid=386, filesize=11.9 K 2023-03-27 04:58:17,525 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:17,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:17,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:17,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:17,529 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 72e4f7cba88d8e5eeeccc51fc07b799f in 862ms, sequenceid=386, compaction requested=true 2023-03-27 04:58:17,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:17,529 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:17,529 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:17,529 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:17,529 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:17,530 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:17,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:17,530 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:17,530 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:17,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:17,530 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,530 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:17,531 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:17,531 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,531 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/B is initiating minor compaction (all files) 2023-03-27 04:58:17,531 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:17,531 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/B in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:17,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:17,531 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:17,531 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/A is initiating minor compaction (all files) 2023-03-27 04:58:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,531 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/361df267d0d64916b610993b1bf66e53, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/c300c369d7f64ef9859cbcec3f71fae5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/2159dd68633c4fc0b9ae6549724eb475] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=36.5 K 2023-03-27 04:58:17,531 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/A in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:17,531 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,531 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/3524cc93c1fa492b8dfcca052f4e2bdf, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/8d1d90d342a7458283c01d8ff9144e6d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/843ba54b49424efc8575d3917e3c3c50] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=36.5 K 2023-03-27 04:58:17,531 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:17,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,532 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 361df267d0d64916b610993b1bf66e53, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1679893094075 2023-03-27 04:58:17,532 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 3524cc93c1fa492b8dfcca052f4e2bdf, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1679893094075 2023-03-27 04:58:17,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:17,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:17,532 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting c300c369d7f64ef9859cbcec3f71fae5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679893094421 2023-03-27 04:58:17,532 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 8d1d90d342a7458283c01d8ff9144e6d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679893094421 2023-03-27 04:58:17,533 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 2159dd68633c4fc0b9ae6549724eb475, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1679893096563 2023-03-27 04:58:17,534 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 843ba54b49424efc8575d3917e3c3c50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1679893096563 2023-03-27 04:58:17,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:17,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:17,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:17,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:17,534 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:17,535 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:17,535 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:17,535 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:17,535 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:17,535 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-27 04:58:17,535 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:17,535 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:17,535 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:17,535 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:17,535 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:17,535 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:17,542 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#B#compaction#442 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:17,543 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#A#compaction#443 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:17,551 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/b7858da3bcb04e4ca259b664dee535f7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/b7858da3bcb04e4ca259b664dee535f7 2023-03-27 04:58:17,554 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/A of 72e4f7cba88d8e5eeeccc51fc07b799f into b7858da3bcb04e4ca259b664dee535f7(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:17,554 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:17,554 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/A, priority=13, startTime=1679893097530; duration=0sec 2023-03-27 04:58:17,554 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:17,554 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:17,555 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:17,555 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/C is initiating minor compaction (all files) 2023-03-27 04:58:17,555 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/C in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:17,555 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/99a6316591914eaa883b72c0c5762854, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/d5a1950fdf2b436faca1b6da5eb16750, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/81b7c0f3c44149dab28c84434d8e77ec] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=36.4 K 2023-03-27 04:58:17,556 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 99a6316591914eaa883b72c0c5762854, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1679893094075 2023-03-27 04:58:17,556 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting d5a1950fdf2b436faca1b6da5eb16750, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679893094421 2023-03-27 04:58:17,556 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 81b7c0f3c44149dab28c84434d8e77ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1679893096563 2023-03-27 04:58:17,561 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#C#compaction#444 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:17,571 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/4a59e689a945461594dc7664ba41074a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/4a59e689a945461594dc7664ba41074a 2023-03-27 04:58:17,576 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/C of 72e4f7cba88d8e5eeeccc51fc07b799f into 4a59e689a945461594dc7664ba41074a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:17,576 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:17,576 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/C, priority=13, startTime=1679893097530; duration=0sec 2023-03-27 04:58:17,576 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:17,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:17,812 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:17,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893157820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893157821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893157823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893157825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893157826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893157927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893157927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893157927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893157929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:17,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893157930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,944 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/694e1f49f4b642ff9b9ab9220f1ba2a7 2023-03-27 04:58:17,948 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/adde3a1646ed419987be04cf56cec2a2 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/adde3a1646ed419987be04cf56cec2a2 2023-03-27 04:58:17,953 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/0db9692771cf4919a9ac786ef71d959a 2023-03-27 04:58:17,954 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/B of 72e4f7cba88d8e5eeeccc51fc07b799f into adde3a1646ed419987be04cf56cec2a2(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:17,954 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:17,954 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/B, priority=13, startTime=1679893097530; duration=0sec 2023-03-27 04:58:17,954 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:17,960 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/5e186e58a1c944cdaf0c7e4a4f2b6ba1 2023-03-27 04:58:17,963 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/694e1f49f4b642ff9b9ab9220f1ba2a7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/694e1f49f4b642ff9b9ab9220f1ba2a7 2023-03-27 04:58:17,965 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/694e1f49f4b642ff9b9ab9220f1ba2a7, entries=150, sequenceid=402, filesize=11.9 K 2023-03-27 04:58:17,966 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/0db9692771cf4919a9ac786ef71d959a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/0db9692771cf4919a9ac786ef71d959a 2023-03-27 04:58:17,968 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/0db9692771cf4919a9ac786ef71d959a, entries=150, sequenceid=402, filesize=11.9 K 2023-03-27 04:58:17,969 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/5e186e58a1c944cdaf0c7e4a4f2b6ba1 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/5e186e58a1c944cdaf0c7e4a4f2b6ba1 2023-03-27 04:58:17,971 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/5e186e58a1c944cdaf0c7e4a4f2b6ba1, entries=150, sequenceid=402, filesize=11.9 K 2023-03-27 04:58:17,972 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 72e4f7cba88d8e5eeeccc51fc07b799f in 437ms, sequenceid=402, compaction requested=false 2023-03-27 04:58:17,972 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:17,972 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:17,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:17,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:17,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:17,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:17,973 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,973 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:17,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:17,973 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,973 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:17,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:17,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:17,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:17,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:17,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45904b2d[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:17,974 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:17,974 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,975 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,975 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:17,975 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:17,975 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:17,975 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,975 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:17,976 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,976 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:17,976 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:17,976 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:17,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:17,979 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:17,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:17,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:17,980 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:17,980 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:17,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:17,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:17,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,981 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:17,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,981 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:17,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:17,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:17,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:17,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,983 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:17,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:17,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,985 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,985 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,986 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:17,985 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:17,986 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:17,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:17,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:17,986 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:17,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:17,986 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:17,986 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:17,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:17,986 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:17,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:17,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:17,986 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:17,986 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:17,986 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:18,086 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:18,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:18,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:18,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:18,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:18,089 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:18,089 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:18,089 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:18,089 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:18,089 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,089 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:18,090 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:18,090 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:18,090 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,090 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:18,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:18,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:18,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:18,091 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:18,091 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:18,091 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:18,091 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:18,091 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:18,091 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:18,091 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:18,091 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:18,091 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-27 04:58:18,092 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:18,092 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:18,092 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:18,092 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:18,092 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:18,092 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:18,098 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/747304a4f1cf4a5b99934894a790cb70 2023-03-27 04:58:18,105 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/8e692dd47c6b497983e95187368e6db1 2023-03-27 04:58:18,112 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/1e26b736da5241dba99e735c4bc3709d 2023-03-27 04:58:18,115 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/747304a4f1cf4a5b99934894a790cb70 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/747304a4f1cf4a5b99934894a790cb70 2023-03-27 04:58:18,117 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/747304a4f1cf4a5b99934894a790cb70, entries=150, sequenceid=421, filesize=11.9 K 2023-03-27 04:58:18,118 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/8e692dd47c6b497983e95187368e6db1 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/8e692dd47c6b497983e95187368e6db1 2023-03-27 04:58:18,121 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/8e692dd47c6b497983e95187368e6db1, entries=150, sequenceid=421, filesize=11.9 K 2023-03-27 04:58:18,121 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/1e26b736da5241dba99e735c4bc3709d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1e26b736da5241dba99e735c4bc3709d 2023-03-27 04:58:18,124 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1e26b736da5241dba99e735c4bc3709d, entries=150, sequenceid=421, filesize=11.9 K 2023-03-27 04:58:18,124 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 72e4f7cba88d8e5eeeccc51fc07b799f in 33ms, sequenceid=421, compaction requested=true 2023-03-27 04:58:18,125 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:18,125 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:18,125 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:18,125 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:18,125 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,125 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:18,125 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:18,126 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:18,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:18,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:18,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dfa3e14[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:18,128 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:18,128 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,129 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,129 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:18,129 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:18,129 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:18,129 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,129 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:18,130 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:18,130 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:18,130 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:18,130 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:18,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:18,134 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:18,134 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:18,134 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:18,134 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:18,134 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:18,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:18,136 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:58:18,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:18,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:18,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:18,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:18,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:18,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:18,142 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,142 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:18,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,142 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:18,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:18,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:18,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:18,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:18,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,143 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:18,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,144 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:18,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,146 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,146 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:18,146 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:18,146 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:18,146 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:18,146 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:18,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:18,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:18,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:18,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:18,146 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,146 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,146 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:18,146 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,147 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:18,147 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:18,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:18,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679893098151, max timestamp=1679893098163Num uniques 859; 2023-03-27 04:58:18,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-27 04:58:18,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679893098151, max timestamp=1679893098163Num uniques 859; 2023-03-27 04:58:18,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-27 04:58:18,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679893098151, max timestamp=1679893098163Num uniques 859; 2023-03-27 04:58:18,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-27 04:58:18,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893158186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893158189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893158190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893158191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893158190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,247 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:18,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:18,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:18,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:18,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:18,249 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:18,249 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:18,249 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:18,249 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:18,249 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,249 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:18,250 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,250 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:18,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:18,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:18,250 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,250 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:18,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:18,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:18,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:18,251 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:18,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:18,251 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:18,251 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:18,251 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:18,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:18,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:18,252 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:18,252 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:18,252 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:18,252 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:18,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:18,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:18,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:18,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:18,253 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:18,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:18,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:18,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:18,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:18,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:18,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:18,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c05b4b0[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:18,254 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:18,254 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,255 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,255 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:18,255 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,255 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:18,255 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:18,255 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:18,256 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:18,256 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:18,256 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,256 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:18,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:18,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:18,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:18,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:18,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:18,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:18,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:18,257 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:18,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:18,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:18,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:18,258 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:18,258 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,258 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:18,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:18,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,258 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:18,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:18,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:18,259 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:18,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:18,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,260 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:18,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,263 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,263 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:18,263 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:18,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:18,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:18,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:18,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:18,263 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:18,263 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:18,263 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:18,264 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:18,264 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,264 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:18,264 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:18,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:18,264 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,264 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893158295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893158298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893158298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893158298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893158298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,364 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:18,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:18,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:18,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:18,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:18,367 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:18,367 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:18,367 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:18,367 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:18,367 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,367 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:18,368 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,368 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:18,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:18,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:18,368 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,368 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:18,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:18,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:18,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:18,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:18,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:18,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:18,369 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:18,369 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:18,369 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:18,369 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:18,369 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:18,369 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:18,369 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. started... 2023-03-27 04:58:18,369 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. as already flushing 2023-03-27 04:58:18,369 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:18,369 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:18,369 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:18,369 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,369 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:18,369 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:18,370 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:18,370 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:18,370 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,370 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:18,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:18,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2db08ba0[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:18,371 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:18,371 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,372 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,372 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:18,372 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:18,372 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:18,372 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,372 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:18,373 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:18,373 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:18,373 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:18,373 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:18,374 DEBUG [Thread-2915] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d7deb76 to 127.0.0.1:61059 2023-03-27 04:58:18,374 DEBUG [Thread-2919] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31fc0e3f to 127.0.0.1:61059 2023-03-27 04:58:18,374 DEBUG [Thread-2915] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:18,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:18,374 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:18,375 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:18,375 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:18,375 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:18,375 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:18,374 DEBUG [Thread-2919] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:18,376 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,376 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:18,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,376 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:18,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:18,377 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,378 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:18,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:18,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:18,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:18,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:18,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,379 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:18,379 DEBUG [Thread-2921] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79db57d4 to 127.0.0.1:61059 2023-03-27 04:58:18,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:18,379 DEBUG [Thread-2921] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:18,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:18,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:18,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:18,379 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:18,380 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:18,380 DEBUG [Thread-2914] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:18,380 DEBUG [Thread-2914] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:18,382 DEBUG [Thread-2923] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e56ea94 to 127.0.0.1:61059 2023-03-27 04:58:18,382 DEBUG [Thread-2923] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:18,382 DEBUG [Thread-2917] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f4b7807 to 127.0.0.1:61059 2023-03-27 04:58:18,383 DEBUG [Thread-2917] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:18,480 DEBUG [Thread-2914] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:18,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:18,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893158503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893158503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893158503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893158503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893158504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,543 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/fae381f2b84249129c519d29e0a8adc4 2023-03-27 04:58:18,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893158806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893158806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893158807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893158807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:18,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893158808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:18,959 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/de06c9789f3249db94bfd20547542c7c 2023-03-27 04:58:19,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:19,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43148 deadline: 1679893159309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:19,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:19,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43102 deadline: 1679893159310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:19,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:19,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43130 deadline: 1679893159310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:19,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:19,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43118 deadline: 1679893159312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:19,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:19,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43142 deadline: 1679893159313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:19,377 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/e8a5d1c2bc7f4f3a962bc28cc9b84227 2023-03-27 04:58:19,385 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/fae381f2b84249129c519d29e0a8adc4 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/fae381f2b84249129c519d29e0a8adc4 2023-03-27 04:58:19,390 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/fae381f2b84249129c519d29e0a8adc4, entries=300, sequenceid=431, filesize=19.0 K 2023-03-27 04:58:19,391 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/de06c9789f3249db94bfd20547542c7c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/de06c9789f3249db94bfd20547542c7c 2023-03-27 04:58:19,397 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/de06c9789f3249db94bfd20547542c7c, entries=150, sequenceid=431, filesize=11.9 K 2023-03-27 04:58:19,397 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/e8a5d1c2bc7f4f3a962bc28cc9b84227 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/e8a5d1c2bc7f4f3a962bc28cc9b84227 2023-03-27 04:58:19,403 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/e8a5d1c2bc7f4f3a962bc28cc9b84227, entries=150, sequenceid=431, filesize=11.9 K 2023-03-27 04:58:19,404 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 72e4f7cba88d8e5eeeccc51fc07b799f in 1268ms, sequenceid=431, compaction requested=true 2023-03-27 04:58:19,404 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:19,404 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:19,404 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:19,404 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:19,404 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:58:19,404 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:58:19,406 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:58:19,406 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/B is initiating minor compaction (all files) 2023-03-27 04:58:19,406 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 56950 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:58:19,406 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/B in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:19,406 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/A is initiating minor compaction (all files) 2023-03-27 04:58:19,407 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/adde3a1646ed419987be04cf56cec2a2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/0db9692771cf4919a9ac786ef71d959a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/8e692dd47c6b497983e95187368e6db1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/de06c9789f3249db94bfd20547542c7c] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=48.5 K 2023-03-27 04:58:19,407 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/A in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:19,407 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/b7858da3bcb04e4ca259b664dee535f7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/694e1f49f4b642ff9b9ab9220f1ba2a7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/747304a4f1cf4a5b99934894a790cb70, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/fae381f2b84249129c519d29e0a8adc4] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=55.6 K 2023-03-27 04:58:19,407 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting adde3a1646ed419987be04cf56cec2a2, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1679893096563 2023-03-27 04:58:19,407 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting b7858da3bcb04e4ca259b664dee535f7, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1679893096563 2023-03-27 04:58:19,408 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 0db9692771cf4919a9ac786ef71d959a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1679893096687 2023-03-27 04:58:19,408 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 694e1f49f4b642ff9b9ab9220f1ba2a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1679893096687 2023-03-27 04:58:19,408 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 8e692dd47c6b497983e95187368e6db1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1679893097822 2023-03-27 04:58:19,408 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 747304a4f1cf4a5b99934894a790cb70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1679893097822 2023-03-27 04:58:19,408 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting de06c9789f3249db94bfd20547542c7c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1679893098132 2023-03-27 04:58:19,409 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting fae381f2b84249129c519d29e0a8adc4, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1679893098131 2023-03-27 04:58:19,419 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#B#compaction#453 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:19,419 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#A#compaction#454 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:19,427 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/2593cdd855d540c195b46e08e3b95724 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/2593cdd855d540c195b46e08e3b95724 2023-03-27 04:58:19,430 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/B of 72e4f7cba88d8e5eeeccc51fc07b799f into 2593cdd855d540c195b46e08e3b95724(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:19,430 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:19,430 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/B, priority=12, startTime=1679893099404; duration=0sec 2023-03-27 04:58:19,430 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:19,430 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:58:19,431 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:58:19,431 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): 72e4f7cba88d8e5eeeccc51fc07b799f/C is initiating minor compaction (all files) 2023-03-27 04:58:19,431 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72e4f7cba88d8e5eeeccc51fc07b799f/C in TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:19,431 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/4a59e689a945461594dc7664ba41074a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/5e186e58a1c944cdaf0c7e4a4f2b6ba1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1e26b736da5241dba99e735c4bc3709d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/e8a5d1c2bc7f4f3a962bc28cc9b84227] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp, totalSize=48.4 K 2023-03-27 04:58:19,431 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 4a59e689a945461594dc7664ba41074a, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1679893096563 2023-03-27 04:58:19,431 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 5e186e58a1c944cdaf0c7e4a4f2b6ba1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1679893096687 2023-03-27 04:58:19,432 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 1e26b736da5241dba99e735c4bc3709d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1679893097822 2023-03-27 04:58:19,432 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting e8a5d1c2bc7f4f3a962bc28cc9b84227, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1679893098132 2023-03-27 04:58:19,438 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): 72e4f7cba88d8e5eeeccc51fc07b799f#C#compaction#455 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:19,836 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/a2ed997d9720454da5e620b7d09df571 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a2ed997d9720454da5e620b7d09df571 2023-03-27 04:58:19,841 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/A of 72e4f7cba88d8e5eeeccc51fc07b799f into a2ed997d9720454da5e620b7d09df571(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:19,841 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:19,841 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/A, priority=12, startTime=1679893099404; duration=0sec 2023-03-27 04:58:19,841 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:19,846 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/cd60048449db4ddfba9a30819e2414d6 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/cd60048449db4ddfba9a30819e2414d6 2023-03-27 04:58:19,851 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 72e4f7cba88d8e5eeeccc51fc07b799f/C of 72e4f7cba88d8e5eeeccc51fc07b799f into cd60048449db4ddfba9a30819e2414d6(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:19,851 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:19,851 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f., storeName=72e4f7cba88d8e5eeeccc51fc07b799f/C, priority=12, startTime=1679893099404; duration=0sec 2023-03-27 04:58:19,851 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:20,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:20,316 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-03-27 04:58:20,316 DEBUG [Thread-2910] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41ccedcb to 127.0.0.1:61059 2023-03-27 04:58:20,316 DEBUG [Thread-2910] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:20,318 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:20,318 DEBUG [Thread-2912] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40e6f624 to 127.0.0.1:61059 2023-03-27 04:58:20,318 DEBUG [Thread-2912] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:20,318 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:20,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:20,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:20,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:20,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:20,321 DEBUG [Thread-2908] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2cb7a788 to 127.0.0.1:61059 2023-03-27 04:58:20,322 DEBUG [Thread-2906] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02e8c6f6 to 127.0.0.1:61059 2023-03-27 04:58:20,322 DEBUG [Thread-2908] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:20,322 DEBUG [Thread-2906] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:20,324 DEBUG [Thread-2904] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a885842 to 127.0.0.1:61059 2023-03-27 04:58:20,324 DEBUG [Thread-2904] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:20,324 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-27 04:58:20,325 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 72 2023-03-27 04:58:20,325 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 76 2023-03-27 04:58:20,325 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 71 2023-03-27 04:58:20,325 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 77 2023-03-27 04:58:20,325 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 76 2023-03-27 04:58:20,325 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-27 04:58:20,325 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-27 04:58:20,325 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(402): scanned 2445 2023-03-27 04:58:20,325 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(403): verified 7330 rows 2023-03-27 04:58:20,325 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(402): scanned 2458 2023-03-27 04:58:20,325 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(403): verified 7373 rows 2023-03-27 04:58:20,325 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(402): scanned 2442 2023-03-27 04:58:20,325 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(403): verified 7326 rows 2023-03-27 04:58:20,325 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(402): scanned 2437 2023-03-27 04:58:20,325 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(403): verified 7311 rows 2023-03-27 04:58:20,325 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(402): scanned 2451 2023-03-27 04:58:20,326 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(403): verified 7353 rows 2023-03-27 04:58:20,326 INFO [Listener at localhost.localdomain/43311] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-27 04:58:20,326 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x09e26d67 to 127.0.0.1:61059 2023-03-27 04:58:20,326 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:20,328 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-27 04:58:20,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-27 04:58:20,329 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/606e168381a841a7862ffd72b89f0180 2023-03-27 04:58:20,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-27 04:58:20,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-27 04:58:20,332 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893100332"}]},"ts":"1679893100332"} 2023-03-27 04:58:20,333 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-27 04:58:20,334 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-27 04:58:20,334 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=72e4f7cba88d8e5eeeccc51fc07b799f, UNASSIGN}] 2023-03-27 04:58:20,335 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=72e4f7cba88d8e5eeeccc51fc07b799f, UNASSIGN 2023-03-27 04:58:20,335 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=72e4f7cba88d8e5eeeccc51fc07b799f, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:20,335 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893100335"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679893100335"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679893100335"}]},"ts":"1679893100335"} 2023-03-27 04:58:20,336 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:58:20,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-27 04:58:20,489 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:20,489 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 72e4f7cba88d8e5eeeccc51fc07b799f, disabling compactions & flushes 2023-03-27 04:58:20,489 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:20,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-27 04:58:20,741 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/260777db2e9d4b0bbfb444e7b7a5abbf 2023-03-27 04:58:20,758 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/1dbae1cd41224eb08017bab2e7768c74 2023-03-27 04:58:20,762 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/606e168381a841a7862ffd72b89f0180 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/606e168381a841a7862ffd72b89f0180 2023-03-27 04:58:20,766 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/606e168381a841a7862ffd72b89f0180, entries=150, sequenceid=471, filesize=11.9 K 2023-03-27 04:58:20,766 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/260777db2e9d4b0bbfb444e7b7a5abbf as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/260777db2e9d4b0bbfb444e7b7a5abbf 2023-03-27 04:58:20,771 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/260777db2e9d4b0bbfb444e7b7a5abbf, entries=150, sequenceid=471, filesize=11.9 K 2023-03-27 04:58:20,771 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/1dbae1cd41224eb08017bab2e7768c74 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1dbae1cd41224eb08017bab2e7768c74 2023-03-27 04:58:20,776 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1dbae1cd41224eb08017bab2e7768c74, entries=150, sequenceid=471, filesize=11.9 K 2023-03-27 04:58:20,777 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=20.13 KB/20610 for 72e4f7cba88d8e5eeeccc51fc07b799f in 461ms, sequenceid=471, compaction requested=false 2023-03-27 04:58:20,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:20,777 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:20,777 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:20,777 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. after waiting 0 ms 2023-03-27 04:58:20,777 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:20,777 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 72e4f7cba88d8e5eeeccc51fc07b799f 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-27 04:58:20,777 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=A 2023-03-27 04:58:20,777 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:20,777 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=B 2023-03-27 04:58:20,777 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:20,777 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72e4f7cba88d8e5eeeccc51fc07b799f, store=C 2023-03-27 04:58:20,778 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:20,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-27 04:58:21,186 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/e1a7e90242284613b3c45e95e956a8e2 2023-03-27 04:58:21,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-27 04:58:21,605 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/76cb0788c23f4261a5c748c8c480e0e4 2023-03-27 04:58:21,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/7f48aebd4ee144a8a3126d501c3d36ca 2023-03-27 04:58:21,627 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/A/e1a7e90242284613b3c45e95e956a8e2 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/e1a7e90242284613b3c45e95e956a8e2 2023-03-27 04:58:21,632 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/e1a7e90242284613b3c45e95e956a8e2, entries=100, sequenceid=477, filesize=9.5 K 2023-03-27 04:58:21,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/B/76cb0788c23f4261a5c748c8c480e0e4 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/76cb0788c23f4261a5c748c8c480e0e4 2023-03-27 04:58:21,639 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/76cb0788c23f4261a5c748c8c480e0e4, entries=100, sequenceid=477, filesize=9.5 K 2023-03-27 04:58:21,640 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/.tmp/C/7f48aebd4ee144a8a3126d501c3d36ca as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/7f48aebd4ee144a8a3126d501c3d36ca 2023-03-27 04:58:21,645 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/7f48aebd4ee144a8a3126d501c3d36ca, entries=100, sequenceid=477, filesize=9.5 K 2023-03-27 04:58:21,646 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 72e4f7cba88d8e5eeeccc51fc07b799f in 869ms, sequenceid=477, compaction requested=true 2023-03-27 04:58:21,668 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a53484e06a5046c389e6f0a46293d015, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a8c23eb358f84989a413ecd6143db071, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/70349533f17a4e21a6a6cfbd95336950, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/c3f800bdb4cd49688498e7b66aa8df00, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/ad12db80f0f948218ecf25bfee5ad2f0, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/5d4ef7d568d14467ab640dd05d9fc481, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/e4ac6734295a45c398e6f945be2be5fb, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/e4f756d520e94f56b294f4c9c8c2b3d2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/c21425d556d4494c9a96059dbbeb4793, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/fbff1d84370f40ec9fa60be20748287e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a6a515094d1244b89febb942c6852182, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/05a0d2c8a35b49e8b6805dd8b370b461, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/dddf239c70ba4ccd864657693462e2c5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/d391527dd9ac4f0c80a6dc0a5308212a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/87c00c1c6e564e7ca5a6867820cf5f08, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/695b1353189a4b039c2f6d0579d03251, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/ecebd15e605b48cc8527b4d1e29fb825, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/20d92b6a753741a3a05da1e2f3a9f43a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/ec1cfcce4a2443669239d674c144fcd8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/9b51dc88efd24f0590f05b180c25eb39, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/bc265aeee051489784054abb58a78ca1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/b54a49a8d040422bbab0feab9b224ffe, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/d0bff69d73324c36ab9d72520f6e902f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/3524cc93c1fa492b8dfcca052f4e2bdf, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/8d1d90d342a7458283c01d8ff9144e6d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/b7858da3bcb04e4ca259b664dee535f7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/843ba54b49424efc8575d3917e3c3c50, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/694e1f49f4b642ff9b9ab9220f1ba2a7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/747304a4f1cf4a5b99934894a790cb70, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/fae381f2b84249129c519d29e0a8adc4] to archive 2023-03-27 04:58:21,669 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:58:21,670 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a53484e06a5046c389e6f0a46293d015 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a53484e06a5046c389e6f0a46293d015 2023-03-27 04:58:21,671 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a8c23eb358f84989a413ecd6143db071 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a8c23eb358f84989a413ecd6143db071 2023-03-27 04:58:21,672 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/70349533f17a4e21a6a6cfbd95336950 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/70349533f17a4e21a6a6cfbd95336950 2023-03-27 04:58:21,673 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/c3f800bdb4cd49688498e7b66aa8df00 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/c3f800bdb4cd49688498e7b66aa8df00 2023-03-27 04:58:21,674 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/ad12db80f0f948218ecf25bfee5ad2f0 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/ad12db80f0f948218ecf25bfee5ad2f0 2023-03-27 04:58:21,675 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/5d4ef7d568d14467ab640dd05d9fc481 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/5d4ef7d568d14467ab640dd05d9fc481 2023-03-27 04:58:21,675 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/e4ac6734295a45c398e6f945be2be5fb to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/e4ac6734295a45c398e6f945be2be5fb 2023-03-27 04:58:21,676 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/e4f756d520e94f56b294f4c9c8c2b3d2 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/e4f756d520e94f56b294f4c9c8c2b3d2 2023-03-27 04:58:21,677 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/c21425d556d4494c9a96059dbbeb4793 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/c21425d556d4494c9a96059dbbeb4793 2023-03-27 04:58:21,678 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/fbff1d84370f40ec9fa60be20748287e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/fbff1d84370f40ec9fa60be20748287e 2023-03-27 04:58:21,679 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a6a515094d1244b89febb942c6852182 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a6a515094d1244b89febb942c6852182 2023-03-27 04:58:21,680 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/05a0d2c8a35b49e8b6805dd8b370b461 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/05a0d2c8a35b49e8b6805dd8b370b461 2023-03-27 04:58:21,680 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/dddf239c70ba4ccd864657693462e2c5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/dddf239c70ba4ccd864657693462e2c5 2023-03-27 04:58:21,681 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/d391527dd9ac4f0c80a6dc0a5308212a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/d391527dd9ac4f0c80a6dc0a5308212a 2023-03-27 04:58:21,682 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/87c00c1c6e564e7ca5a6867820cf5f08 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/87c00c1c6e564e7ca5a6867820cf5f08 2023-03-27 04:58:21,683 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/695b1353189a4b039c2f6d0579d03251 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/695b1353189a4b039c2f6d0579d03251 2023-03-27 04:58:21,685 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/ecebd15e605b48cc8527b4d1e29fb825 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/ecebd15e605b48cc8527b4d1e29fb825 2023-03-27 04:58:21,686 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/20d92b6a753741a3a05da1e2f3a9f43a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/20d92b6a753741a3a05da1e2f3a9f43a 2023-03-27 04:58:21,687 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/ec1cfcce4a2443669239d674c144fcd8 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/ec1cfcce4a2443669239d674c144fcd8 2023-03-27 04:58:21,688 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/9b51dc88efd24f0590f05b180c25eb39 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/9b51dc88efd24f0590f05b180c25eb39 2023-03-27 04:58:21,689 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/bc265aeee051489784054abb58a78ca1 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/bc265aeee051489784054abb58a78ca1 2023-03-27 04:58:21,690 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/b54a49a8d040422bbab0feab9b224ffe to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/b54a49a8d040422bbab0feab9b224ffe 2023-03-27 04:58:21,691 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/d0bff69d73324c36ab9d72520f6e902f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/d0bff69d73324c36ab9d72520f6e902f 2023-03-27 04:58:21,692 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/3524cc93c1fa492b8dfcca052f4e2bdf to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/3524cc93c1fa492b8dfcca052f4e2bdf 2023-03-27 04:58:21,693 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/8d1d90d342a7458283c01d8ff9144e6d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/8d1d90d342a7458283c01d8ff9144e6d 2023-03-27 04:58:21,694 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/b7858da3bcb04e4ca259b664dee535f7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/b7858da3bcb04e4ca259b664dee535f7 2023-03-27 04:58:21,696 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/843ba54b49424efc8575d3917e3c3c50 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/843ba54b49424efc8575d3917e3c3c50 2023-03-27 04:58:21,697 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/694e1f49f4b642ff9b9ab9220f1ba2a7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/694e1f49f4b642ff9b9ab9220f1ba2a7 2023-03-27 04:58:21,698 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/747304a4f1cf4a5b99934894a790cb70 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/747304a4f1cf4a5b99934894a790cb70 2023-03-27 04:58:21,699 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/fae381f2b84249129c519d29e0a8adc4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/fae381f2b84249129c519d29e0a8adc4 2023-03-27 04:58:21,704 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/7d320b4903ed49b79c751e1cb7a5a995, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/e7889c114fd3434692dc5371c61ea742, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/082fc3a8016f44ad8708adaaf1b3cc02, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/815406931936497ea1f481d4389046f5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/c58379f2d71a4d6793318f94b8ec528f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/54e2e6fa1f664f03a9cf945be7c2cf88, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/200b1ee7baa84e1d8788e114601ead9e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/af06c26221754aa6a0cf7027358f9aac, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/a5208cdf43804ced9e4bb95b496e5527, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/559a36d43a6a47f1881671cc519f659d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/5a90307443f94509a1034861c90ffad3, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/42f23daf5fc8477facf527608d651db4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/f20ec49f44e042389f76a4328ebf2e8c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/bab189c8c3264e71be3ea9cc51afdfb6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/2c3b9abc92ec4b248e25cef9f683e53e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/8e2c756acdcc4e3d93bfbeeef1eff684, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/eba19d7f6d304698b7eccdfd931463c8, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/ebfc1d35233f447b8481b5c36117c19a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/17183f4335694e079bcd96563c387ef6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/fcb05b2041404e3eb30b5654cfcd0b03, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/00e3ff56049141eaa826ca03f749d3e9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/3bcc7e036c574c2aa98282d6b97b700c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/361df267d0d64916b610993b1bf66e53, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/70f733bf8a254d52895ba353ffeade73, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/c300c369d7f64ef9859cbcec3f71fae5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/adde3a1646ed419987be04cf56cec2a2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/2159dd68633c4fc0b9ae6549724eb475, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/0db9692771cf4919a9ac786ef71d959a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/8e692dd47c6b497983e95187368e6db1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/de06c9789f3249db94bfd20547542c7c] to archive 2023-03-27 04:58:21,705 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:58:21,706 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/7d320b4903ed49b79c751e1cb7a5a995 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/7d320b4903ed49b79c751e1cb7a5a995 2023-03-27 04:58:21,707 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/e7889c114fd3434692dc5371c61ea742 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/e7889c114fd3434692dc5371c61ea742 2023-03-27 04:58:21,708 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/082fc3a8016f44ad8708adaaf1b3cc02 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/082fc3a8016f44ad8708adaaf1b3cc02 2023-03-27 04:58:21,709 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/815406931936497ea1f481d4389046f5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/815406931936497ea1f481d4389046f5 2023-03-27 04:58:21,710 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/c58379f2d71a4d6793318f94b8ec528f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/c58379f2d71a4d6793318f94b8ec528f 2023-03-27 04:58:21,711 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/54e2e6fa1f664f03a9cf945be7c2cf88 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/54e2e6fa1f664f03a9cf945be7c2cf88 2023-03-27 04:58:21,711 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/200b1ee7baa84e1d8788e114601ead9e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/200b1ee7baa84e1d8788e114601ead9e 2023-03-27 04:58:21,712 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/af06c26221754aa6a0cf7027358f9aac to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/af06c26221754aa6a0cf7027358f9aac 2023-03-27 04:58:21,713 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/a5208cdf43804ced9e4bb95b496e5527 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/a5208cdf43804ced9e4bb95b496e5527 2023-03-27 04:58:21,714 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/559a36d43a6a47f1881671cc519f659d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/559a36d43a6a47f1881671cc519f659d 2023-03-27 04:58:21,714 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/5a90307443f94509a1034861c90ffad3 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/5a90307443f94509a1034861c90ffad3 2023-03-27 04:58:21,715 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/42f23daf5fc8477facf527608d651db4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/42f23daf5fc8477facf527608d651db4 2023-03-27 04:58:21,715 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/f20ec49f44e042389f76a4328ebf2e8c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/f20ec49f44e042389f76a4328ebf2e8c 2023-03-27 04:58:21,716 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/bab189c8c3264e71be3ea9cc51afdfb6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/bab189c8c3264e71be3ea9cc51afdfb6 2023-03-27 04:58:21,717 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/2c3b9abc92ec4b248e25cef9f683e53e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/2c3b9abc92ec4b248e25cef9f683e53e 2023-03-27 04:58:21,717 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/8e2c756acdcc4e3d93bfbeeef1eff684 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/8e2c756acdcc4e3d93bfbeeef1eff684 2023-03-27 04:58:21,718 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/eba19d7f6d304698b7eccdfd931463c8 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/eba19d7f6d304698b7eccdfd931463c8 2023-03-27 04:58:21,719 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/ebfc1d35233f447b8481b5c36117c19a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/ebfc1d35233f447b8481b5c36117c19a 2023-03-27 04:58:21,719 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/17183f4335694e079bcd96563c387ef6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/17183f4335694e079bcd96563c387ef6 2023-03-27 04:58:21,720 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/fcb05b2041404e3eb30b5654cfcd0b03 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/fcb05b2041404e3eb30b5654cfcd0b03 2023-03-27 04:58:21,721 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/00e3ff56049141eaa826ca03f749d3e9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/00e3ff56049141eaa826ca03f749d3e9 2023-03-27 04:58:21,721 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/3bcc7e036c574c2aa98282d6b97b700c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/3bcc7e036c574c2aa98282d6b97b700c 2023-03-27 04:58:21,722 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/361df267d0d64916b610993b1bf66e53 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/361df267d0d64916b610993b1bf66e53 2023-03-27 04:58:21,723 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/70f733bf8a254d52895ba353ffeade73 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/70f733bf8a254d52895ba353ffeade73 2023-03-27 04:58:21,724 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/c300c369d7f64ef9859cbcec3f71fae5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/c300c369d7f64ef9859cbcec3f71fae5 2023-03-27 04:58:21,724 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/adde3a1646ed419987be04cf56cec2a2 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/adde3a1646ed419987be04cf56cec2a2 2023-03-27 04:58:21,725 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/2159dd68633c4fc0b9ae6549724eb475 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/2159dd68633c4fc0b9ae6549724eb475 2023-03-27 04:58:21,726 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/0db9692771cf4919a9ac786ef71d959a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/0db9692771cf4919a9ac786ef71d959a 2023-03-27 04:58:21,727 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/8e692dd47c6b497983e95187368e6db1 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/8e692dd47c6b497983e95187368e6db1 2023-03-27 04:58:21,728 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/de06c9789f3249db94bfd20547542c7c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/de06c9789f3249db94bfd20547542c7c 2023-03-27 04:58:21,731 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/168feefd03634291b02e629e9c8404d3, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/8927d160474246b7ad1c460ab26eab58, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/c8aad07a200847d592082c699cd4c101, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/45bf3280e00b4561ae8bfe123b5fedda, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/4f8894ce90034c14a1a7bb88dd8ff974, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1f2e632d223f4ecdbcc2d7bd40012dad, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1dc39bd6c94a4a50948a6bb4607a440c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1391d8a0cbcf4efebb598e31a4e9e6c7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/e187132e77004c73bb0d7e2332020247, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/10d42a4b3e7746e382933c0a4787dd4b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/3c4f6f6839f34de18b3b0be7ed9dce1d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/c28e24c316924de89d91425bcfaba933, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/5b9398f767a949b4966b7705bb7c6cd0, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/f496c7da19864989ba87741d50697355, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/81c694c12a364045bfeb7adc470575f6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/da4c427c71084eecbc33a1b70f8f9d5f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/00320c6322954030a84c6618d8b5c341, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/a7ce4726b25e4f27aea578f8488e508f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/3f006ac3af3e4739ae3f4eb25460e0ba, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/9b55aa9fb33a4196acc130c407e2994a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/2c381f23393b4a7f984d70e593f50fdb, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/99a6316591914eaa883b72c0c5762854, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/6af1b2fd78654e098de613732b9664aa, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/d5a1950fdf2b436faca1b6da5eb16750, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/4a59e689a945461594dc7664ba41074a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/81b7c0f3c44149dab28c84434d8e77ec, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/5e186e58a1c944cdaf0c7e4a4f2b6ba1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1e26b736da5241dba99e735c4bc3709d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/e8a5d1c2bc7f4f3a962bc28cc9b84227] to archive 2023-03-27 04:58:21,732 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:58:21,733 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/168feefd03634291b02e629e9c8404d3 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/168feefd03634291b02e629e9c8404d3 2023-03-27 04:58:21,734 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/8927d160474246b7ad1c460ab26eab58 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/8927d160474246b7ad1c460ab26eab58 2023-03-27 04:58:21,735 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/c8aad07a200847d592082c699cd4c101 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/c8aad07a200847d592082c699cd4c101 2023-03-27 04:58:21,736 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/45bf3280e00b4561ae8bfe123b5fedda to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/45bf3280e00b4561ae8bfe123b5fedda 2023-03-27 04:58:21,737 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/4f8894ce90034c14a1a7bb88dd8ff974 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/4f8894ce90034c14a1a7bb88dd8ff974 2023-03-27 04:58:21,739 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1f2e632d223f4ecdbcc2d7bd40012dad to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1f2e632d223f4ecdbcc2d7bd40012dad 2023-03-27 04:58:21,740 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1dc39bd6c94a4a50948a6bb4607a440c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1dc39bd6c94a4a50948a6bb4607a440c 2023-03-27 04:58:21,741 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1391d8a0cbcf4efebb598e31a4e9e6c7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1391d8a0cbcf4efebb598e31a4e9e6c7 2023-03-27 04:58:21,742 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/e187132e77004c73bb0d7e2332020247 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/e187132e77004c73bb0d7e2332020247 2023-03-27 04:58:21,743 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/10d42a4b3e7746e382933c0a4787dd4b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/10d42a4b3e7746e382933c0a4787dd4b 2023-03-27 04:58:21,745 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/3c4f6f6839f34de18b3b0be7ed9dce1d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/3c4f6f6839f34de18b3b0be7ed9dce1d 2023-03-27 04:58:21,746 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/c28e24c316924de89d91425bcfaba933 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/c28e24c316924de89d91425bcfaba933 2023-03-27 04:58:21,747 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/5b9398f767a949b4966b7705bb7c6cd0 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/5b9398f767a949b4966b7705bb7c6cd0 2023-03-27 04:58:21,748 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/f496c7da19864989ba87741d50697355 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/f496c7da19864989ba87741d50697355 2023-03-27 04:58:21,749 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/81c694c12a364045bfeb7adc470575f6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/81c694c12a364045bfeb7adc470575f6 2023-03-27 04:58:21,751 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/da4c427c71084eecbc33a1b70f8f9d5f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/da4c427c71084eecbc33a1b70f8f9d5f 2023-03-27 04:58:21,752 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/00320c6322954030a84c6618d8b5c341 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/00320c6322954030a84c6618d8b5c341 2023-03-27 04:58:21,754 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/a7ce4726b25e4f27aea578f8488e508f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/a7ce4726b25e4f27aea578f8488e508f 2023-03-27 04:58:21,755 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/3f006ac3af3e4739ae3f4eb25460e0ba to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/3f006ac3af3e4739ae3f4eb25460e0ba 2023-03-27 04:58:21,757 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/9b55aa9fb33a4196acc130c407e2994a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/9b55aa9fb33a4196acc130c407e2994a 2023-03-27 04:58:21,759 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/2c381f23393b4a7f984d70e593f50fdb to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/2c381f23393b4a7f984d70e593f50fdb 2023-03-27 04:58:21,760 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/99a6316591914eaa883b72c0c5762854 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/99a6316591914eaa883b72c0c5762854 2023-03-27 04:58:21,762 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/6af1b2fd78654e098de613732b9664aa to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/6af1b2fd78654e098de613732b9664aa 2023-03-27 04:58:21,764 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/d5a1950fdf2b436faca1b6da5eb16750 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/d5a1950fdf2b436faca1b6da5eb16750 2023-03-27 04:58:21,765 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/4a59e689a945461594dc7664ba41074a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/4a59e689a945461594dc7664ba41074a 2023-03-27 04:58:21,766 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/81b7c0f3c44149dab28c84434d8e77ec to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/81b7c0f3c44149dab28c84434d8e77ec 2023-03-27 04:58:21,768 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/5e186e58a1c944cdaf0c7e4a4f2b6ba1 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/5e186e58a1c944cdaf0c7e4a4f2b6ba1 2023-03-27 04:58:21,769 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1e26b736da5241dba99e735c4bc3709d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1e26b736da5241dba99e735c4bc3709d 2023-03-27 04:58:21,770 DEBUG [StoreCloser-TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/e8a5d1c2bc7f4f3a962bc28cc9b84227 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/e8a5d1c2bc7f4f3a962bc28cc9b84227 2023-03-27 04:58:21,774 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/recovered.edits/480.seqid, newMaxSeqId=480, maxSeqId=1 2023-03-27 04:58:21,775 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f. 2023-03-27 04:58:21,775 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 72e4f7cba88d8e5eeeccc51fc07b799f: 2023-03-27 04:58:21,777 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:21,777 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=72e4f7cba88d8e5eeeccc51fc07b799f, regionState=CLOSED 2023-03-27 04:58:21,777 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893101777"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679893101777"}]},"ts":"1679893101777"} 2023-03-27 04:58:21,779 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-27 04:58:21,779 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 72e4f7cba88d8e5eeeccc51fc07b799f, server=jenkins-hbase20.apache.org,44923,1679892955505 in 1.4420 sec 2023-03-27 04:58:21,780 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-27 04:58:21,780 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=72e4f7cba88d8e5eeeccc51fc07b799f, UNASSIGN in 1.4450 sec 2023-03-27 04:58:21,781 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893101780"}]},"ts":"1679893101780"} 2023-03-27 04:58:21,781 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-27 04:58:21,782 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-27 04:58:21,783 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.4550 sec 2023-03-27 04:58:22,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-27 04:58:22,440 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-27 04:58:22,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-27 04:58:22,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:58:22,445 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:58:22,446 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:58:22,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-27 04:58:22,450 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:22,452 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/recovered.edits] 2023-03-27 04:58:22,454 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/606e168381a841a7862ffd72b89f0180 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/606e168381a841a7862ffd72b89f0180 2023-03-27 04:58:22,455 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a2ed997d9720454da5e620b7d09df571 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/a2ed997d9720454da5e620b7d09df571 2023-03-27 04:58:22,456 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/e1a7e90242284613b3c45e95e956a8e2 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/A/e1a7e90242284613b3c45e95e956a8e2 2023-03-27 04:58:22,458 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/2593cdd855d540c195b46e08e3b95724 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/2593cdd855d540c195b46e08e3b95724 2023-03-27 04:58:22,459 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/260777db2e9d4b0bbfb444e7b7a5abbf to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/260777db2e9d4b0bbfb444e7b7a5abbf 2023-03-27 04:58:22,460 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/76cb0788c23f4261a5c748c8c480e0e4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/B/76cb0788c23f4261a5c748c8c480e0e4 2023-03-27 04:58:22,462 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1dbae1cd41224eb08017bab2e7768c74 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/1dbae1cd41224eb08017bab2e7768c74 2023-03-27 04:58:22,464 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/7f48aebd4ee144a8a3126d501c3d36ca to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/7f48aebd4ee144a8a3126d501c3d36ca 2023-03-27 04:58:22,465 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/cd60048449db4ddfba9a30819e2414d6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/C/cd60048449db4ddfba9a30819e2414d6 2023-03-27 04:58:22,468 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/recovered.edits/480.seqid to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f/recovered.edits/480.seqid 2023-03-27 04:58:22,468 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/72e4f7cba88d8e5eeeccc51fc07b799f 2023-03-27 04:58:22,468 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-27 04:58:22,471 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:58:22,472 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-27 04:58:22,473 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-27 04:58:22,474 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:58:22,475 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-27 04:58:22,475 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679893102475"}]},"ts":"9223372036854775807"} 2023-03-27 04:58:22,476 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-27 04:58:22,476 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 72e4f7cba88d8e5eeeccc51fc07b799f, NAME => 'TestAcidGuarantees,,1679893076127.72e4f7cba88d8e5eeeccc51fc07b799f.', STARTKEY => '', ENDKEY => ''}] 2023-03-27 04:58:22,476 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-27 04:58:22,477 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679893102476"}]},"ts":"9223372036854775807"} 2023-03-27 04:58:22,478 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-27 04:58:22,479 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:58:22,481 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 38 msec 2023-03-27 04:58:22,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-27 04:58:22,548 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-27 04:58:22,565 INFO [Listener at localhost.localdomain/43311] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=230 (was 227) - Thread LEAK? -, OpenFileDescriptor=561 (was 531) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=416 (was 412) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=992 (was 878) - AvailableMemoryMB LEAK? - 2023-03-27 04:58:22,579 INFO [Listener at localhost.localdomain/43311] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=230, OpenFileDescriptor=561, MaxFileDescriptor=60000, SystemLoadAverage=416, ProcessCount=167, AvailableMemoryMB=992 2023-03-27 04:58:22,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-27 04:58:22,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-27 04:58:22,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-27 04:58:22,585 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-27 04:58:22,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-27 04:58:22,585 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-27 04:58:22,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-27 04:58:22,586 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:22,587 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a empty. 2023-03-27 04:58:22,587 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:22,587 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-27 04:58:22,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-27 04:58:22,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-27 04:58:22,996 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-27 04:58:22,998 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a34144f135b39d6005ad383002fabe3a, NAME => 'TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp 2023-03-27 04:58:23,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-27 04:58:23,407 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:58:23,407 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a34144f135b39d6005ad383002fabe3a, disabling compactions & flushes 2023-03-27 04:58:23,407 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:23,407 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:23,407 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. after waiting 0 ms 2023-03-27 04:58:23,407 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:23,407 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:23,407 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:23,411 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-27 04:58:23,413 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893103413"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679893103413"}]},"ts":"1679893103413"} 2023-03-27 04:58:23,415 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-27 04:58:23,417 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-27 04:58:23,417 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893103417"}]},"ts":"1679893103417"} 2023-03-27 04:58:23,419 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-27 04:58:23,423 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a34144f135b39d6005ad383002fabe3a, ASSIGN}] 2023-03-27 04:58:23,425 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a34144f135b39d6005ad383002fabe3a, ASSIGN 2023-03-27 04:58:23,427 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a34144f135b39d6005ad383002fabe3a, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,44923,1679892955505; forceNewPlan=false, retain=false 2023-03-27 04:58:23,579 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=a34144f135b39d6005ad383002fabe3a, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:23,579 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893103579"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679893103579"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679893103579"}]},"ts":"1679893103579"} 2023-03-27 04:58:23,582 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:58:23,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-27 04:58:23,742 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:23,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a34144f135b39d6005ad383002fabe3a, NAME => 'TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.', STARTKEY => '', ENDKEY => ''} 2023-03-27 04:58:23,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:23,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:58:23,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:23,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:23,746 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:23,749 DEBUG [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A 2023-03-27 04:58:23,749 DEBUG [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A 2023-03-27 04:58:23,749 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:58:23,749 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a34144f135b39d6005ad383002fabe3a columnFamilyName A 2023-03-27 04:58:23,750 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.HStore(310): Store=a34144f135b39d6005ad383002fabe3a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:58:23,750 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:23,753 DEBUG [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B 2023-03-27 04:58:23,753 DEBUG [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B 2023-03-27 04:58:23,753 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:58:23,753 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a34144f135b39d6005ad383002fabe3a columnFamilyName B 2023-03-27 04:58:23,754 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.HStore(310): Store=a34144f135b39d6005ad383002fabe3a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:58:23,755 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:23,757 DEBUG [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C 2023-03-27 04:58:23,757 DEBUG [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C 2023-03-27 04:58:23,757 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:58:23,757 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a34144f135b39d6005ad383002fabe3a columnFamilyName C 2023-03-27 04:58:23,758 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.HStore(310): Store=a34144f135b39d6005ad383002fabe3a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:58:23,758 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:23,759 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:23,759 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:23,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-27 04:58:23,762 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:23,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-27 04:58:23,765 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened a34144f135b39d6005ad383002fabe3a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67875596, jitterRate=0.011425197124481201}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-27 04:58:23,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:23,765 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., pid=56, masterSystemTime=1679893103735 2023-03-27 04:58:23,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:23,767 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:23,767 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=a34144f135b39d6005ad383002fabe3a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:23,767 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893103767"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679893103767"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679893103767"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679893103767"}]},"ts":"1679893103767"} 2023-03-27 04:58:23,769 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-27 04:58:23,769 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 in 186 msec 2023-03-27 04:58:23,771 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-27 04:58:23,771 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a34144f135b39d6005ad383002fabe3a, ASSIGN in 347 msec 2023-03-27 04:58:23,772 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-27 04:58:23,772 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893103772"}]},"ts":"1679893103772"} 2023-03-27 04:58:23,773 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-27 04:58:23,774 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-27 04:58:23,776 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1920 sec 2023-03-27 04:58:24,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-27 04:58:24,694 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-27 04:58:24,695 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x0b200ba7 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:58:24,701 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a5c51ec, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:58:24,703 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:58:24,704 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50882, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:58:24,706 DEBUG [Listener at localhost.localdomain/43311] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-27 04:58:24,709 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57128, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-27 04:58:24,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-27 04:58:24,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-27 04:58:24,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-27 04:58:25,121 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-27 04:58:25,121 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-27 04:58:25,122 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-27 04:58:25,123 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-27 04:58:25,127 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a34144f135b39d6005ad383002fabe3a, REOPEN/MOVE}] 2023-03-27 04:58:25,128 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a34144f135b39d6005ad383002fabe3a, REOPEN/MOVE 2023-03-27 04:58:25,128 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=a34144f135b39d6005ad383002fabe3a, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,128 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893105128"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679893105128"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679893105128"}]},"ts":"1679893105128"} 2023-03-27 04:58:25,129 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:58:25,281 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:25,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing a34144f135b39d6005ad383002fabe3a, disabling compactions & flushes 2023-03-27 04:58:25,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:25,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:25,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. after waiting 0 ms 2023-03-27 04:58:25,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:25,285 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-27 04:58:25,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:25,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:25,286 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: a34144f135b39d6005ad383002fabe3a to self. 2023-03-27 04:58:25,288 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:25,288 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=a34144f135b39d6005ad383002fabe3a, regionState=CLOSED 2023-03-27 04:58:25,288 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893105288"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679893105288"}]},"ts":"1679893105288"} 2023-03-27 04:58:25,291 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-27 04:58:25,291 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 in 161 msec 2023-03-27 04:58:25,292 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a34144f135b39d6005ad383002fabe3a, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,44923,1679892955505; forceNewPlan=false, retain=true 2023-03-27 04:58:25,442 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=a34144f135b39d6005ad383002fabe3a, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,443 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893105442"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679893105442"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679893105442"}]},"ts":"1679893105442"} 2023-03-27 04:58:25,444 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:58:25,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:25,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a34144f135b39d6005ad383002fabe3a, NAME => 'TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.', STARTKEY => '', ENDKEY => ''} 2023-03-27 04:58:25,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:25,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-27 04:58:25,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:25,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:25,601 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:25,602 DEBUG [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A 2023-03-27 04:58:25,602 DEBUG [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A 2023-03-27 04:58:25,602 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:58:25,602 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a34144f135b39d6005ad383002fabe3a columnFamilyName A 2023-03-27 04:58:25,604 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.HStore(310): Store=a34144f135b39d6005ad383002fabe3a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:58:25,604 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:25,605 DEBUG [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B 2023-03-27 04:58:25,605 DEBUG [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B 2023-03-27 04:58:25,605 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:58:25,605 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a34144f135b39d6005ad383002fabe3a columnFamilyName B 2023-03-27 04:58:25,606 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.HStore(310): Store=a34144f135b39d6005ad383002fabe3a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:58:25,606 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:25,606 DEBUG [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C 2023-03-27 04:58:25,606 DEBUG [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C 2023-03-27 04:58:25,607 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-27 04:58:25,607 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a34144f135b39d6005ad383002fabe3a columnFamilyName C 2023-03-27 04:58:25,607 INFO [StoreOpener-a34144f135b39d6005ad383002fabe3a-1] regionserver.HStore(310): Store=a34144f135b39d6005ad383002fabe3a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-27 04:58:25,607 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:25,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:25,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:25,610 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-27 04:58:25,611 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:25,612 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened a34144f135b39d6005ad383002fabe3a; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64162815, jitterRate=-0.043899551033973694}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-27 04:58:25,612 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:25,612 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., pid=61, masterSystemTime=1679893105596 2023-03-27 04:58:25,614 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=a34144f135b39d6005ad383002fabe3a, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,614 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:25,614 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893105614"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679893105614"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679893105614"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679893105614"}]},"ts":"1679893105614"} 2023-03-27 04:58:25,614 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:25,616 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-27 04:58:25,616 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 in 171 msec 2023-03-27 04:58:25,617 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-27 04:58:25,617 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a34144f135b39d6005ad383002fabe3a, REOPEN/MOVE in 489 msec 2023-03-27 04:58:25,618 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-27 04:58:25,619 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 495 msec 2023-03-27 04:58:25,619 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 908 msec 2023-03-27 04:58:25,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-27 04:58:25,621 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x50131d14 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:58:25,625 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5be55982, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:58:25,625 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x6722b207 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:58:25,629 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f11e84a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:58:25,629 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x48502121 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:58:25,632 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a1010a6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:58:25,633 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x08d3d8bc to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:58:25,636 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4143b9ae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:58:25,637 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x7e6ad437 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:58:25,642 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@686af57e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:58:25,643 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x078cd331 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:58:25,651 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@387217bb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:58:25,652 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x1d74fc48 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:58:25,660 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c00a18c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:58:25,661 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x070ab5f5 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:58:25,684 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3da68fe0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:58:25,685 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x2daeb795 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:58:25,693 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7fd90ac8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:58:25,693 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(139): Connect 0x28ac2b09 to 127.0.0.1:61059 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-27 04:58:25,702 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e63396, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-27 04:58:25,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:25,722 DEBUG [hconnection-0x757ba870-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:58:25,723 DEBUG [hconnection-0x14db1382-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:58:25,723 DEBUG [hconnection-0x6136cc8c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:58:25,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:25,726 DEBUG [hconnection-0x2146da81-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:58:25,731 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50888, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:58:25,724 DEBUG [hconnection-0x3ee88a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:58:25,724 DEBUG [hconnection-0x66d8edeb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:58:25,734 DEBUG [hconnection-0x7aa983d1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:58:25,733 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50896, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:58:25,732 DEBUG [hconnection-0x156dbdf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:58:25,731 DEBUG [hconnection-0x5a282e0a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:58:25,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:25,736 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:25,736 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:25,736 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:25,736 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:25,736 DEBUG [hconnection-0x294e2dae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-27 04:58:25,736 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50910, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:58:25,737 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50926, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:58:25,738 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50962, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:58:25,739 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,739 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:25,740 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,740 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:25,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:25,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:25,740 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,740 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:25,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:25,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:25,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:25,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:25,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:25,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:25,744 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:25,744 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:25,744 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:25,748 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:25,750 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:25,750 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:25,752 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50914, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:58:25,752 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50936, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:58:25,752 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50974, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:58:25,750 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:25,752 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50916, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:58:25,756 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:25,757 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:25,757 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:25,757 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:25,757 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,757 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:25,757 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:25,757 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50948, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-27 04:58:25,765 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:25,765 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:25,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:25,766 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:25,766 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:25,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:25,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:25,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:25,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:25,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:25,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a3e6418[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:25,768 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:25,768 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:25,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:25,769 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:58:25,769 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:25,769 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:25,769 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:25,770 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:25,770 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:25,770 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:25,770 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:25,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:25,770 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:25,770 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:25,770 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:25,771 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:25,771 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:25,771 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:25,771 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,772 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:25,772 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:25,772 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:25,772 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:25,772 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:25,772 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:25,772 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:25,773 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,773 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:25,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:25,773 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:25,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:25,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:25,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:25,774 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,776 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,778 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:25,778 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:25,778 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:25,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:25,778 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:25,778 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:25,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:25,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:25,778 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:25,779 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,779 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:25,779 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:25,779 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,779 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:25,779 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:25,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:25,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:25,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:25,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679893105776, max timestamp=1679893105789Num uniques 859; 2023-03-27 04:58:25,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-27 04:58:25,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679893105776, max timestamp=1679893105789Num uniques 859; 2023-03-27 04:58:25,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679893105776, max timestamp=1679893105789Num uniques 859; 2023-03-27 04:58:25,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-27 04:58:25,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-27 04:58:25,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:25,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893165796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:25,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893165796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:25,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893165799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:25,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893165800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:25,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893165800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,879 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:25,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:25,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:25,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:25,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:25,882 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:25,882 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:25,882 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:25,882 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:25,882 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,882 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:25,883 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,883 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:25,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:25,883 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,883 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:25,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:25,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:25,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:25,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:25,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:25,884 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:25,884 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:25,884 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:25,884 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:25,884 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:25,885 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:25,885 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:25,885 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:25,885 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:25,885 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:25,885 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:25,885 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,885 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:25,885 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:25,886 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,886 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:25,886 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:25,886 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:25,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:25,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26f8fdd5[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:25,888 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:25,888 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:25,889 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:25,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:25,889 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:25,889 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:25,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:25,889 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:25,889 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,889 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:25,889 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:25,890 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:25,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:25,892 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:25,892 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:25,892 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:25,892 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:25,892 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:25,892 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,892 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:25,893 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:25,893 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:25,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,894 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:25,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,895 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,895 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:25,895 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:25,895 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:25,895 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:25,895 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:25,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:25,895 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:25,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:25,895 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:25,896 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:25,896 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:25,896 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:25,896 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:25,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:25,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:25,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893165902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:25,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893165902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:25,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893165904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:25,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893165904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:25,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893165904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:25,996 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:25,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:25,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:26,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:26,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:26,000 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:26,000 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,000 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:26,000 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:26,000 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,000 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,001 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:26,001 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,001 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,001 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:26,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:26,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:26,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:26,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:26,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:26,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:26,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:26,002 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,002 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:26,002 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:26,002 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:26,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:26,002 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,003 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:26,003 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:26,003 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,003 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,004 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,004 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:26,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17e97d8e[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:26,005 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:26,005 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,005 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,005 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:26,005 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:26,005 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:26,005 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,005 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:26,006 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:26,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:26,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:26,008 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:26,008 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:26,008 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:26,008 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:26,008 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:26,009 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,009 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,009 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:26,010 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,011 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,013 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:26,013 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,014 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:26,014 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:26,014 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:26,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:26,014 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:26,014 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,014 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,014 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,014 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:26,014 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:26,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:26,014 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:26,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:26,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:26,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893166103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:26,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893166103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:26,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893166105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:26,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893166106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:26,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893166106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,114 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:26,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:26,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:26,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:26,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:26,117 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:26,117 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,117 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:26,117 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:26,117 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,117 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,118 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:26,118 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:26,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:26,118 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,118 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:26,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:26,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:26,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:26,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:26,119 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:26,119 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,119 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:26,119 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:26,119 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:26,119 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,119 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:26,119 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:26,119 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,119 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:26,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:26,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:26,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:26,120 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,120 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,121 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,121 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:26,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ead1958[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:26,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:26,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,122 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,122 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:26,123 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:26,123 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,123 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:26,123 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:26,123 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,123 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:26,123 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:26,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,123 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:26,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:26,125 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:26,125 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:26,125 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:26,125 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:26,125 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:26,126 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,126 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,126 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,127 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,131 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,142 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:26,142 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,143 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:26,143 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:26,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:26,143 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:26,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:26,143 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:26,143 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:26,143 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,143 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:26,143 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,143 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,143 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:26,190 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327c6f9c97f3783440fa433ad8c0953a2f6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327c6f9c97f3783440fa433ad8c0953a2f6 2023-03-27 04:58:26,243 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:26,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:26,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:26,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:26,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:26,246 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:26,246 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,246 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:26,246 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:26,246 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,246 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,247 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:26,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:26,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:26,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:26,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:26,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:26,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:26,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:26,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:26,248 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,248 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:26,248 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:26,248 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:26,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:26,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:26,249 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:26,249 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,249 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,249 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,249 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:26,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@272ab431[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:26,250 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:26,250 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,251 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,251 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:26,251 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:26,251 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:26,251 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,251 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:26,252 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:26,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:26,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:26,253 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:26,254 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:26,254 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:26,254 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:26,254 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:26,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,254 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:26,255 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,256 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,257 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:26,257 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:26,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:26,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:26,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:26,257 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:26,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:26,257 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,257 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:26,258 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:26,258 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:26,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:26,258 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,258 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,258 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,258 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,358 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:26,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:26,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:26,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:26,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:26,360 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:26,360 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,360 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:26,361 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:26,361 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,361 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,362 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:26,362 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:26,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:26,362 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,362 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:26,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:26,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:26,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:26,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:26,363 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:26,363 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,363 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:26,363 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:26,363 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:26,363 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,363 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:26,363 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:26,363 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,363 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:26,363 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:26,363 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,363 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:26,363 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:26,364 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,364 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,365 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,365 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:26,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@465df354[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:26,366 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:26,366 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,366 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,366 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:26,366 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:26,366 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:26,367 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,367 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:26,367 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:26,367 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:26,367 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,367 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:26,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:26,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:26,369 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:26,369 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:26,369 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:26,369 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:26,369 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:26,370 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,370 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:26,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,370 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,370 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,371 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,372 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:26,372 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:26,372 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:26,373 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:26,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:26,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:26,373 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,373 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,373 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,373 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,373 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:26,373 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,373 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,373 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:26,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:26,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:26,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:26,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893166404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:26,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893166407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:26,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893166409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:26,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893166409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:26,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893166410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,473 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:26,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:26,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:26,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:26,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:26,476 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:26,476 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,476 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:26,476 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:26,476 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,476 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,476 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:26,476 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:26,477 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,477 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:26,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:26,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:26,478 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:26,478 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,478 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:26,478 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:26,478 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:26,478 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,478 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:26,478 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:26,478 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,478 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:26,478 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:26,478 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,478 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:26,478 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:26,480 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,480 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,480 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,480 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:26,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:26,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a04f31c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:26,481 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:26,481 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,482 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,482 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:26,482 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:26,482 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:26,482 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,482 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:26,482 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:26,483 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:26,483 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:26,482 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:26,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:26,484 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:26,484 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:26,484 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:26,484 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:26,484 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:26,485 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,485 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,486 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:26,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:26,486 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,487 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,488 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,488 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,488 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:26,488 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:26,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:26,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:26,488 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:26,488 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:26,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:26,488 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,489 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:26,489 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,489 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:26,489 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,489 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:26,589 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:26,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:26,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:26,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:26,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:26,592 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:26,592 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,592 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:26,592 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:26,592 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,592 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,593 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,593 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:26,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:26,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:26,593 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,593 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:26,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:26,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:26,595 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:26,595 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,595 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:26,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:26,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:26,598 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/2151ffec0fe440f2acb7fedb6033c90a 2023-03-27 04:58:26,598 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,598 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:26,598 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:26,598 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:26,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:26,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:26,598 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:26,599 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,599 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,600 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,600 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:26,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cb54edd[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:26,601 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:26,601 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,602 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,602 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:26,602 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:26,602 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:26,602 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,602 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:26,603 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:26,604 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:26,604 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:26,604 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:26,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:26,606 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:26,606 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:26,607 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:26,607 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:26,607 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:26,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:26,607 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,608 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,610 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,615 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:26,615 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:26,615 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,615 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,615 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,615 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:26,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:26,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:26,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:26,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:26,616 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:26,616 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:26,615 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,616 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,616 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,616 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:26,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:26,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:26,716 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:26,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:26,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:26,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:26,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:26,719 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:26,719 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,719 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:26,719 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:26,719 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,719 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,720 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:26,720 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:26,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:26,721 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,721 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:26,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:26,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:26,722 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:26,722 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,722 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:26,722 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:26,723 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:26,723 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,723 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:26,723 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:26,723 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,723 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:26,723 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:26,723 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,723 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:26,723 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:26,724 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,725 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,725 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:26,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:26,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e4ce046[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:26,726 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:26,726 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,726 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,726 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,727 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:26,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:26,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:26,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:26,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:26,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:26,727 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:26,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:26,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:26,729 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:26,729 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:26,729 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:26,729 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:26,729 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:26,730 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,730 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,730 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:26,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:26,731 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,747 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,749 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:26,749 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:26,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:26,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:26,749 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,749 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,749 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,749 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,749 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,749 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,749 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,749 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:26,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:26,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:26,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:26,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:26,750 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:26,750 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:26,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:26,850 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:26,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:26,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:26,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:26,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:26,852 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:26,852 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,852 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:26,852 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:26,853 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,853 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,854 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:26,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:26,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,854 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:26,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:26,855 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:26,855 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,855 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:26,855 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:26,855 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:26,855 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,855 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:26,855 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:26,855 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,855 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:26,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:26,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:26,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:26,856 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,856 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,857 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:26,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:26,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17226388[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:26,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:26,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,861 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,861 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:26,861 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:26,861 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:26,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:26,862 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,862 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:26,862 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:26,862 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:26,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:26,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:26,864 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:26,864 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:26,864 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:26,864 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:26,864 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:26,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:26,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,865 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,866 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,867 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,869 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:26,869 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:26,869 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,869 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:26,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:26,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:26,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:26,869 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,870 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,870 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:26,870 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,870 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:26,870 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:26,870 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,870 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:26,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893166910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:26,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893166911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:26,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893166912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:26,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893166912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:26,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893166915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,970 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:26,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:26,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:26,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:26,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:26,978 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:26,978 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,978 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:26,978 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:26,978 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,978 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,979 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,979 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:26,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:26,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:26,979 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,979 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:26,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:26,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:26,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:26,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:26,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:26,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:26,986 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:26,990 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:26,990 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:26,990 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,990 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:26,990 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:26,990 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:26,990 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:26,990 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:26,990 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,990 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:26,990 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:26,991 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,991 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,991 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,992 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:26,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a83ee63[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:26,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:26,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,993 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:26,993 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:26,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:26,993 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:26,993 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:26,994 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,994 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:26,994 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:26,994 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:26,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:26,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:26,998 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:26,998 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:26,998 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:26,998 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:26,998 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:26,999 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,999 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,999 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:26,999 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:27,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:27,000 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:27,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:27,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:27,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:27,002 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:27,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:27,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:27,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:27,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:27,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:27,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:27,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:27,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:27,002 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:27,002 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:27,002 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:27,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:27,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:27,020 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/065c94b100dd4400bcabe249a6d62178 2023-03-27 04:58:27,035 WARN [DataStreamer for file /user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/c5989fc4514e44e0a7e88a12806eed09] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-03-27 04:58:27,036 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/c5989fc4514e44e0a7e88a12806eed09 2023-03-27 04:58:27,039 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/2151ffec0fe440f2acb7fedb6033c90a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/2151ffec0fe440f2acb7fedb6033c90a 2023-03-27 04:58:27,042 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/2151ffec0fe440f2acb7fedb6033c90a, entries=150, sequenceid=16, filesize=25.1 K 2023-03-27 04:58:27,043 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/065c94b100dd4400bcabe249a6d62178 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/065c94b100dd4400bcabe249a6d62178 2023-03-27 04:58:27,046 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/065c94b100dd4400bcabe249a6d62178, entries=150, sequenceid=16, filesize=11.6 K 2023-03-27 04:58:27,046 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/c5989fc4514e44e0a7e88a12806eed09 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/c5989fc4514e44e0a7e88a12806eed09 2023-03-27 04:58:27,049 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/c5989fc4514e44e0a7e88a12806eed09, entries=150, sequenceid=16, filesize=11.6 K 2023-03-27 04:58:27,050 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a34144f135b39d6005ad383002fabe3a in 1280ms, sequenceid=16, compaction requested=false 2023-03-27 04:58:27,050 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:27,102 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:27,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:27,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:27,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:27,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:27,105 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:27,105 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:27,105 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:27,105 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:27,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:27,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:27,106 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:27,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:27,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:27,106 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:27,111 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:27,111 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:27,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:27,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:27,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:27,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:27,112 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:27,112 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:27,112 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:27,112 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:27,113 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:27,113 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:27,113 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:27,113 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-27 04:58:27,113 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:27,113 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:27,113 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:27,113 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:27,113 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:27,113 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:27,128 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303274984819bdbad4aca991d567e74464b5b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303274984819bdbad4aca991d567e74464b5b 2023-03-27 04:58:27,302 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-27 04:58:27,548 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/ef03f4089e9946a5891f9408aa2ae999 2023-03-27 04:58:27,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:27,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:27,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:27,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893167924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:27,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:27,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893167924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:27,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:27,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893167926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:27,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:27,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893167926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:27,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:27,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893167927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:27,979 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/4bfaed941f4d4c9095e27635ad7c63b7 2023-03-27 04:58:28,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893168028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893168028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893168030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893168030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893168031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893168231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893168232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893168232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893168232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893168238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,392 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/ce834cc136ea4d2c8bba09c4179bc6a9 2023-03-27 04:58:28,394 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/ef03f4089e9946a5891f9408aa2ae999 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/ef03f4089e9946a5891f9408aa2ae999 2023-03-27 04:58:28,397 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/ef03f4089e9946a5891f9408aa2ae999, entries=150, sequenceid=50, filesize=25.1 K 2023-03-27 04:58:28,397 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/4bfaed941f4d4c9095e27635ad7c63b7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/4bfaed941f4d4c9095e27635ad7c63b7 2023-03-27 04:58:28,407 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/4bfaed941f4d4c9095e27635ad7c63b7, entries=150, sequenceid=50, filesize=11.6 K 2023-03-27 04:58:28,410 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/ce834cc136ea4d2c8bba09c4179bc6a9 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ce834cc136ea4d2c8bba09c4179bc6a9 2023-03-27 04:58:28,413 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ce834cc136ea4d2c8bba09c4179bc6a9, entries=150, sequenceid=50, filesize=11.6 K 2023-03-27 04:58:28,414 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for a34144f135b39d6005ad383002fabe3a in 1301ms, sequenceid=50, compaction requested=false 2023-03-27 04:58:28,414 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:28,414 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:28,414 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:28,414 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:28,414 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,414 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:28,414 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:28,415 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,415 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:28,415 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,415 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:28,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:28,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c268e48[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:28,417 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:28,417 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,417 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,417 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,418 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:28,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:28,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:28,418 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:28,419 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:28,419 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:28,419 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:28,419 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:28,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:28,421 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:28,421 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:28,421 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:28,421 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:28,421 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:28,422 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,422 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:28,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,422 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:28,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:28,423 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,424 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,425 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,425 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,425 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,425 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,425 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:28,425 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:28,425 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:28,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:28,426 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,426 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:28,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:28,426 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,426 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:28,426 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:28,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:28,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:28,526 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:28,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:28,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:28,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:28,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:28,537 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:28,538 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:28,538 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:28,538 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:28,538 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:28,538 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,538 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-27 04:58:28,539 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:28,539 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:28,539 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,539 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:28,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:28,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:28,542 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:28,542 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:28,542 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:28,543 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:28,543 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:28,543 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:28,543 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:28,543 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:28,543 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:28,543 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:28,543 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:28,543 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,543 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:28,543 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:28,546 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:28,546 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:28,546 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:28,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:28,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:28,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:28,551 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,551 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:28,551 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,551 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:28,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:28,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:28,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15b1195a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:28,554 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:28,554 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,559 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,559 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:28,559 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:28,559 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:28,560 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,560 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:28,563 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303274aecf9e7782641e78cad3e2248552467 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303274aecf9e7782641e78cad3e2248552467 2023-03-27 04:58:28,567 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/7b8b2d649b8a49f6aedc8cd948e1e531 2023-03-27 04:58:28,568 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:28,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893168563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:28,568 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:28,568 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:28,568 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:28,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893168564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893168567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893168567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:28,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893168569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,571 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:28,571 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:28,571 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:28,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:28,572 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:28,572 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:28,576 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,576 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:28,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,576 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:28,577 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:28,579 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:28,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,584 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,584 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,584 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,584 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,584 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,584 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:28,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:28,584 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:28,584 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:28,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:28,584 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,584 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:28,585 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:28,585 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:28,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:28,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:28,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893168669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893168670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893168670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893168670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893168672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,685 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:28,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:28,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:28,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:28,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:28,691 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:28,691 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:28,691 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:28,691 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:28,691 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,691 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,692 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:28,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:28,692 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,693 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,693 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:28,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:28,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:28,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:28,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:28,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:28,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:28,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:28,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:28,694 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:28,694 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:28,694 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:28,697 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:28,697 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:28,697 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:28,698 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,698 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:28,698 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:28,701 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,701 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:28,701 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:28,701 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:28,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:28,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@118b1647[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:28,703 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:28,703 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,703 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:28,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:28,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:28,704 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,704 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:28,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:28,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:28,704 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:28,704 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:28,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:28,707 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:28,707 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:28,707 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:28,707 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:28,707 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:28,714 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,714 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:28,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,714 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:28,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,715 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:28,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,716 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,726 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,726 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,726 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,726 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:28,726 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:28,726 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:28,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:28,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:28,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:28,726 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:28,726 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,727 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,727 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,727 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:28,727 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:28,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:28,827 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:28,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:28,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:28,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:28,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:28,837 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:28,837 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:28,837 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:28,837 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:28,838 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,838 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:28,839 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:28,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:28,839 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,839 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:28,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:28,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:28,841 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:28,841 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:28,841 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:28,841 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:28,841 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:28,841 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:28,841 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:28,841 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:28,841 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:28,841 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:28,841 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:28,842 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,842 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:28,842 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:28,843 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,843 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:28,845 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,845 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:28,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:28,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@244e02e3[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:28,848 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:28,848 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,851 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,851 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,852 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:28,852 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:28,852 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:28,852 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:28,859 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,859 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:28,859 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:28,860 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:28,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:28,862 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:28,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:28,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:28,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:28,863 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:28,863 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,863 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:28,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,863 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:28,864 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:28,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,865 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,866 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:28,866 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,866 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,866 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:28,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:28,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:28,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:28,866 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:28,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:28,866 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:28,866 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:28,866 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,866 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,866 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:28,867 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,867 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:28,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:28,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893168872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893168873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893168874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893168874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:28,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893168874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,967 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:28,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:28,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:28,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:28,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:28,974 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:28,974 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:28,974 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:28,974 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:28,974 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,975 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,975 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:28,975 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:28,975 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,976 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:28,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:28,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:28,978 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:28,978 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:28,978 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:28,978 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/ac96204ceea448c8afbcefe480c607da 2023-03-27 04:58:28,980 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:28,980 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:28,980 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:28,980 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:28,980 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:28,980 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:28,980 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:28,980 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:28,980 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,980 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:28,980 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:28,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,984 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:28,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,984 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:28,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:28,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c628cec[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:28,987 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:28,987 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,992 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:28,993 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:28,993 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:28,993 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:28,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,993 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:28,993 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:28,993 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,993 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:28,994 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:28,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:28,995 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:28,995 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:28,995 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:28,996 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:28,996 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:28,997 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,997 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:28,997 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:28,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:28,998 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:28,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:28,999 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:28,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,000 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,000 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,000 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,000 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,000 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,000 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,001 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,001 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:29,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:29,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:29,001 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:29,001 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:29,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:29,001 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:29,001 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/b9192da1c3a64076ac9a983c7af770cb 2023-03-27 04:58:29,001 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:29,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:29,005 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/7b8b2d649b8a49f6aedc8cd948e1e531 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/7b8b2d649b8a49f6aedc8cd948e1e531 2023-03-27 04:58:29,008 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/7b8b2d649b8a49f6aedc8cd948e1e531, entries=150, sequenceid=63, filesize=25.1 K 2023-03-27 04:58:29,009 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/ac96204ceea448c8afbcefe480c607da as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/ac96204ceea448c8afbcefe480c607da 2023-03-27 04:58:29,012 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/ac96204ceea448c8afbcefe480c607da, entries=150, sequenceid=63, filesize=11.6 K 2023-03-27 04:58:29,012 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/b9192da1c3a64076ac9a983c7af770cb as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b9192da1c3a64076ac9a983c7af770cb 2023-03-27 04:58:29,015 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b9192da1c3a64076ac9a983c7af770cb, entries=150, sequenceid=63, filesize=11.6 K 2023-03-27 04:58:29,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for a34144f135b39d6005ad383002fabe3a in 478ms, sequenceid=63, compaction requested=true 2023-03-27 04:58:29,016 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:29,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:29,016 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:29,016 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:29,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:29,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:29,017 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:29,017 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:29,017 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/A is initiating minor compaction (all files) 2023-03-27 04:58:29,017 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/B is initiating minor compaction (all files) 2023-03-27 04:58:29,017 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/A in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:29,017 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/B in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:29,017 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/2151ffec0fe440f2acb7fedb6033c90a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/ef03f4089e9946a5891f9408aa2ae999, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/7b8b2d649b8a49f6aedc8cd948e1e531] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=75.4 K 2023-03-27 04:58:29,017 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/065c94b100dd4400bcabe249a6d62178, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/4bfaed941f4d4c9095e27635ad7c63b7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/ac96204ceea448c8afbcefe480c607da] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=34.8 K 2023-03-27 04:58:29,018 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 065c94b100dd4400bcabe249a6d62178, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679893105765 2023-03-27 04:58:29,018 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 2151ffec0fe440f2acb7fedb6033c90a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679893105765 2023-03-27 04:58:29,018 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 4bfaed941f4d4c9095e27635ad7c63b7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679893105798 2023-03-27 04:58:29,018 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting ef03f4089e9946a5891f9408aa2ae999, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679893105798 2023-03-27 04:58:29,018 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting ac96204ceea448c8afbcefe480c607da, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679893107925 2023-03-27 04:58:29,018 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 7b8b2d649b8a49f6aedc8cd948e1e531, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679893107925 2023-03-27 04:58:29,023 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#B#compaction#471 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:29,027 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#A#compaction#472 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:29,031 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/a0e314b3af98485e9d5bec29c5e9c459 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/a0e314b3af98485e9d5bec29c5e9c459 2023-03-27 04:58:29,034 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a34144f135b39d6005ad383002fabe3a/B of a34144f135b39d6005ad383002fabe3a into a0e314b3af98485e9d5bec29c5e9c459(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:29,034 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:29,034 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/B, priority=13, startTime=1679893109016; duration=0sec 2023-03-27 04:58:29,034 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:29,034 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:29,035 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:29,035 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/C is initiating minor compaction (all files) 2023-03-27 04:58:29,035 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/C in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:29,035 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/c5989fc4514e44e0a7e88a12806eed09, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ce834cc136ea4d2c8bba09c4179bc6a9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b9192da1c3a64076ac9a983c7af770cb] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=34.8 K 2023-03-27 04:58:29,035 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting c5989fc4514e44e0a7e88a12806eed09, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679893105765 2023-03-27 04:58:29,035 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting ce834cc136ea4d2c8bba09c4179bc6a9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679893105798 2023-03-27 04:58:29,036 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting b9192da1c3a64076ac9a983c7af770cb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679893107925 2023-03-27 04:58:29,039 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#C#compaction#473 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:29,101 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:29,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:29,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:29,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:29,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:29,104 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:29,104 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:29,104 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:29,104 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:29,104 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,104 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,105 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:29,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:29,105 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,105 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,105 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:29,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:29,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,106 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:29,106 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:29,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:29,106 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:29,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:29,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:29,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:29,107 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:29,107 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:29,107 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:29,107 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-27 04:58:29,107 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:29,107 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:29,107 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:29,107 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:29,107 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:29,107 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:29,136 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303275f30812cc1444c808499c9613e124a2c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303275f30812cc1444c808499c9613e124a2c 2023-03-27 04:58:29,141 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/c1d2d6cd788c472591d611811df08ef7 2023-03-27 04:58:29,149 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/4e9f632b057e4e3e8f90d5d907f55a4e 2023-03-27 04:58:29,175 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/3cb5dd7214a74be9bc9a79053d6f4bf4 2023-03-27 04:58:29,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:29,178 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:29,180 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/c1d2d6cd788c472591d611811df08ef7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/c1d2d6cd788c472591d611811df08ef7 2023-03-27 04:58:29,183 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/c1d2d6cd788c472591d611811df08ef7, entries=150, sequenceid=83, filesize=25.1 K 2023-03-27 04:58:29,184 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/4e9f632b057e4e3e8f90d5d907f55a4e as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/4e9f632b057e4e3e8f90d5d907f55a4e 2023-03-27 04:58:29,187 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/4e9f632b057e4e3e8f90d5d907f55a4e, entries=150, sequenceid=83, filesize=11.6 K 2023-03-27 04:58:29,187 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/3cb5dd7214a74be9bc9a79053d6f4bf4 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/3cb5dd7214a74be9bc9a79053d6f4bf4 2023-03-27 04:58:29,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893169184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893169184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893169185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893169185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893169187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,191 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/3cb5dd7214a74be9bc9a79053d6f4bf4, entries=150, sequenceid=83, filesize=11.6 K 2023-03-27 04:58:29,191 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for a34144f135b39d6005ad383002fabe3a in 84ms, sequenceid=83, compaction requested=false 2023-03-27 04:58:29,191 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:29,191 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:29,191 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:29,191 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:29,191 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,192 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:29,192 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:29,193 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,193 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,193 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:29,194 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:29,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@410425f3[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:29,200 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:29,200 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,209 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,209 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:29,209 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:29,209 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:29,209 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,210 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:29,210 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,210 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:29,210 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,210 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:29,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:29,213 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:29,213 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:29,213 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:29,213 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:29,213 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:29,214 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,214 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,214 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,215 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,217 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,218 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:29,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:29,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:29,219 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:29,219 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:29,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:29,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:29,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:29,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:29,219 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:29,219 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:29,222 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,222 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,222 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,222 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,222 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,222 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,222 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:29,292 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-27 04:58:29,293 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:29,293 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:29,293 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:29,293 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:29,293 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:29,293 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:29,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893169300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893169302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893169303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,307 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893169303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,307 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893169304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,319 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:29,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:29,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:29,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:29,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:29,322 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:29,322 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:29,322 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:29,322 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:29,322 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,322 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,323 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:29,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:29,323 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,323 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,323 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:29,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:29,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:29,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:29,324 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:29,324 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:29,324 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:29,324 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:29,324 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:29,324 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:29,324 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:29,324 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:29,324 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:29,324 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:29,324 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:29,324 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,324 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:29,324 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:29,325 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,325 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,325 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,325 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:29,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8f42aba[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:29,332 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:29,332 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,334 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:29,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:29,334 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,334 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:29,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:29,335 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:29,335 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:29,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:29,337 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:29,337 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:29,337 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:29,337 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:29,337 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:29,337 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,337 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,337 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,338 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,339 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:29,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:29,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:29,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:29,340 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:29,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:29,340 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:29,340 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:29,341 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:29,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:29,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:29,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:29,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893169405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893169407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893169407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893169408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893169408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,441 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:29,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:29,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:29,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:29,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:29,446 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:29,446 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:29,446 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:29,446 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:29,446 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,446 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,448 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:29,448 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:29,448 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/70addde566d8431b8af5d1725ea5e202 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/70addde566d8431b8af5d1725ea5e202 2023-03-27 04:58:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,448 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,449 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:29,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:29,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:29,449 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:29,449 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:29,449 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:29,450 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:29,450 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:29,450 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:29,450 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:29,450 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:29,450 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:29,451 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:29,451 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:29,451 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,451 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:29,451 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:29,452 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,452 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,452 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,452 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,453 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a34144f135b39d6005ad383002fabe3a/C of a34144f135b39d6005ad383002fabe3a into 70addde566d8431b8af5d1725ea5e202(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:29,453 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:29,453 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/C, priority=13, startTime=1679893109016; duration=0sec 2023-03-27 04:58:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,453 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:29,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c79685d[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:29,454 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:29,455 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,455 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,455 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:29,455 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:29,455 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,455 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:29,455 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:29,456 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,456 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,456 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:29,456 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:29,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:29,458 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:29,458 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:29,458 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:29,458 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:29,458 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:29,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:29,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,459 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:29,460 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,461 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,463 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:29,463 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:29,463 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:29,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:29,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:29,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:29,463 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:29,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:29,463 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,463 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:29,463 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,463 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:29,464 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,464 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,464 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,464 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,564 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:29,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:29,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:29,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:29,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:29,571 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:29,571 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:29,571 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:29,571 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:29,572 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,572 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,573 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:29,573 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:29,573 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,573 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:29,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:29,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:29,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:29,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:29,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:29,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:29,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:29,578 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:29,578 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:29,578 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:29,578 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:29,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:29,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:29,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:29,579 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:29,584 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,584 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,590 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,590 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:29,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:29,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14b49ad2[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:29,592 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:29,592 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,593 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,593 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:29,593 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:29,593 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:29,593 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,593 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:29,595 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,595 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:29,595 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:29,595 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:29,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:29,599 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:29,599 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:29,599 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:29,599 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:29,599 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:29,601 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,601 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:29,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,601 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,603 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,604 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,609 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,609 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,609 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,609 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,609 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:29,609 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:29,609 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:29,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:29,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:29,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:29,609 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,610 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:29,610 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,610 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:29,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:29,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893169610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893169611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893169611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893169612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893169612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,702 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303275aa4c9096f07476abec818270e763e22 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303275aa4c9096f07476abec818270e763e22 2023-03-27 04:58:29,710 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:29,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:29,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:29,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:29,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:29,712 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:29,712 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:29,713 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:29,713 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:29,713 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,713 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,713 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:29,713 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:29,714 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,714 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,715 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:29,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:29,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:29,715 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:29,715 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:29,715 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:29,715 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:29,715 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:29,715 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:29,715 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:29,715 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:29,715 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:29,715 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:29,715 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:29,715 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,715 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:29,715 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:29,716 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:29,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@747f0e0a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:29,717 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:29,717 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,718 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,718 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:29,718 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:29,718 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:29,718 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,718 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:29,718 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:29,718 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:29,718 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,718 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:29,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:29,720 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:29,720 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:29,720 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:29,720 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:29,720 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:29,721 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,721 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,721 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:29,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,721 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:29,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:29,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,723 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,723 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,723 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,723 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:29,723 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:29,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:29,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:29,723 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:29,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,724 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:29,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:29,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:29,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:29,724 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:29,724 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:29,824 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:29,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:29,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:29,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:29,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:29,827 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:29,827 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:29,827 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:29,827 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:29,827 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,827 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,828 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,828 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:29,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:29,828 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,828 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:29,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:29,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:29,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:29,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:29,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:29,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:29,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:29,829 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:29,829 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:29,829 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:29,829 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:29,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:29,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:29,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,830 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:29,830 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:29,830 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,830 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,831 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,831 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:29,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:29,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d774bc5[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:29,832 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:29,832 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,832 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,832 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:29,832 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:29,832 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:29,832 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,832 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:29,833 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:29,833 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,833 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,833 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:29,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:29,835 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:29,835 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:29,835 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:29,835 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:29,835 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:29,836 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,836 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,836 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,836 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,838 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:29,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:29,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:29,840 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:29,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:29,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:29,839 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:29,840 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:29,839 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:29,840 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:29,840 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,840 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893169912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893169914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893169915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893169916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:29,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893169916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,940 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:29,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:29,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:29,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:29,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:29,943 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:29,943 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:29,943 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:29,944 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:29,944 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,944 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,945 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:29,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:29,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:29,946 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,946 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,946 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:29,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:29,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:29,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:29,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:29,947 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:29,947 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:29,947 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:29,947 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:29,947 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:29,947 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:29,947 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:29,947 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:29,947 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:29,947 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:29,947 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:29,947 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,947 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:29,947 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:29,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,948 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,948 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,948 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:29,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:29,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a5bfbb6[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:29,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:29,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,951 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,952 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:29,952 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:29,952 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:29,954 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,955 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:29,955 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:29,955 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:29,955 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:29,955 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:29,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:29,957 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:29,957 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:29,957 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:29,957 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:29,957 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:29,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:29,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,958 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:29,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:29,959 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,960 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:29,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:29,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:29,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:29,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:29,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:29,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:29,962 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:29,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:29,962 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:29,962 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:29,962 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:29,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:29,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:30,062 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:30,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:30,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:30,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:30,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:30,065 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:30,065 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:30,065 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:30,065 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:30,078 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:30,078 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:30,079 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:30,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:30,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:30,079 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,079 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,079 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:30,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:30,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:30,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:30,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:30,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:30,080 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:30,080 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:30,080 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:30,082 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:30,082 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:30,086 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:30,086 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:30,086 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:30,086 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:30,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:30,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:30,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:30,086 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:30,092 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:30,092 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:30,093 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:30,093 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:30,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:30,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a899a25[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:30,094 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:30,094 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:30,101 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:30,101 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,101 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:30,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:30,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:30,101 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:30,102 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:30,106 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=100, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/fcbe22d7477f490dba0efbe26e76f4d4 2023-03-27 04:58:30,109 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:30,109 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:30,109 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:30,109 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:30,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:30,112 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:30,112 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:30,112 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:30,112 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:30,112 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:30,117 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:30,118 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:30,118 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:30,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:30,119 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:30,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,130 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,131 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/dbd87b621fa7488da717bd351a1dd193 2023-03-27 04:58:30,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:30,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:30,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:30,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:30,132 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:30,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:30,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:30,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:30,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:30,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:30,133 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:30,133 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:30,133 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:30,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:30,145 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/b6ec2edc216f45a4bd81164c2bce3d66 2023-03-27 04:58:30,150 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/fcbe22d7477f490dba0efbe26e76f4d4 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/fcbe22d7477f490dba0efbe26e76f4d4 2023-03-27 04:58:30,153 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/fcbe22d7477f490dba0efbe26e76f4d4, entries=250, sequenceid=100, filesize=38.7 K 2023-03-27 04:58:30,154 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/dbd87b621fa7488da717bd351a1dd193 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/dbd87b621fa7488da717bd351a1dd193 2023-03-27 04:58:30,157 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/dbd87b621fa7488da717bd351a1dd193, entries=150, sequenceid=100, filesize=11.6 K 2023-03-27 04:58:30,157 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/b6ec2edc216f45a4bd81164c2bce3d66 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b6ec2edc216f45a4bd81164c2bce3d66 2023-03-27 04:58:30,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b6ec2edc216f45a4bd81164c2bce3d66, entries=150, sequenceid=100, filesize=11.6 K 2023-03-27 04:58:30,161 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for a34144f135b39d6005ad383002fabe3a in 869ms, sequenceid=100, compaction requested=true 2023-03-27 04:58:30,161 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:30,161 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:30,161 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:30,161 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-03-27 04:58:30,161 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-27 04:58:30,161 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-27 04:58:30,162 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-27 04:58:30,162 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. because compaction request was cancelled 2023-03-27 04:58:30,162 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:30,162 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:30,162 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/B is initiating minor compaction (all files) 2023-03-27 04:58:30,162 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/B in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:30,162 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/a0e314b3af98485e9d5bec29c5e9c459, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/4e9f632b057e4e3e8f90d5d907f55a4e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/dbd87b621fa7488da717bd351a1dd193] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=34.9 K 2023-03-27 04:58:30,163 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting a0e314b3af98485e9d5bec29c5e9c459, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679893107925 2023-03-27 04:58:30,163 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 4e9f632b057e4e3e8f90d5d907f55a4e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679893108562 2023-03-27 04:58:30,163 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting dbd87b621fa7488da717bd351a1dd193, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1679893109185 2023-03-27 04:58:30,167 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#B#compaction#480 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:30,233 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:30,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:30,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:30,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:30,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:30,243 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:30,243 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:30,243 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:30,243 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:30,244 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:30,244 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:30,245 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:30,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:30,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:30,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:30,246 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:30,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:30,247 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/283e844b2d1c4cd99de1fd67592124cc as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/283e844b2d1c4cd99de1fd67592124cc 2023-03-27 04:58:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:30,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:30,252 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a34144f135b39d6005ad383002fabe3a/A of a34144f135b39d6005ad383002fabe3a into 283e844b2d1c4cd99de1fd67592124cc(size=25.2 K), total size for store is 89.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-27 04:58:30,252 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:30,252 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/A, priority=13, startTime=1679893109016; duration=1sec 2023-03-27 04:58:30,252 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:30,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:30,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:30,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:30,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:30,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:30,252 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:30,254 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:30,254 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/C is initiating minor compaction (all files) 2023-03-27 04:58:30,254 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/C in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:30,254 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/70addde566d8431b8af5d1725ea5e202, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/3cb5dd7214a74be9bc9a79053d6f4bf4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b6ec2edc216f45a4bd81164c2bce3d66] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=34.9 K 2023-03-27 04:58:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:30,254 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:30,254 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:30,254 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-27 04:58:30,254 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 70addde566d8431b8af5d1725ea5e202, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679893107925 2023-03-27 04:58:30,255 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:30,255 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 3cb5dd7214a74be9bc9a79053d6f4bf4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679893108562 2023-03-27 04:58:30,255 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:30,255 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:30,255 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:30,255 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:30,256 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:30,256 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting b6ec2edc216f45a4bd81164c2bce3d66, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1679893109185 2023-03-27 04:58:30,265 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#C#compaction#482 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:30,298 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303270bde693f7628497a9cca732f4bc509f6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303270bde693f7628497a9cca732f4bc509f6 2023-03-27 04:58:30,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:30,420 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:30,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:30,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893170433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:30,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893170436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:30,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893170438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:30,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893170439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:30,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893170442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:30,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893170546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:30,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893170546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:30,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893170547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:30,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893170547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:30,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893170547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,577 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/b492350e1347453d94d85a33b0d0d5bd as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/b492350e1347453d94d85a33b0d0d5bd 2023-03-27 04:58:30,582 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a34144f135b39d6005ad383002fabe3a/B of a34144f135b39d6005ad383002fabe3a into b492350e1347453d94d85a33b0d0d5bd(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:30,582 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:30,582 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/B, priority=13, startTime=1679893110161; duration=0sec 2023-03-27 04:58:30,583 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:30,702 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/c04df40d242e461ea799653acf1e7650 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/c04df40d242e461ea799653acf1e7650 2023-03-27 04:58:30,710 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a34144f135b39d6005ad383002fabe3a/C of a34144f135b39d6005ad383002fabe3a into c04df40d242e461ea799653acf1e7650(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:30,710 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:30,710 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/C, priority=13, startTime=1679893110161; duration=0sec 2023-03-27 04:58:30,710 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:30,724 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=117, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/86be74830c85442dbda54c9fde901971 2023-03-27 04:58:30,746 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/7584a31f199846d39ce97150800eca50 2023-03-27 04:58:30,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:30,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893170750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:30,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893170750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:30,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:30,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893170751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893170751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:30,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:30,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893170751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893171054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893171054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893171055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893171056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893171057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,208 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/fa67c5ca641f47f5a58017c7acbb1c39 2023-03-27 04:58:31,213 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/86be74830c85442dbda54c9fde901971 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/86be74830c85442dbda54c9fde901971 2023-03-27 04:58:31,231 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/86be74830c85442dbda54c9fde901971, entries=150, sequenceid=117, filesize=25.1 K 2023-03-27 04:58:31,231 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/7584a31f199846d39ce97150800eca50 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/7584a31f199846d39ce97150800eca50 2023-03-27 04:58:31,243 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/7584a31f199846d39ce97150800eca50, entries=150, sequenceid=117, filesize=11.6 K 2023-03-27 04:58:31,244 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/fa67c5ca641f47f5a58017c7acbb1c39 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/fa67c5ca641f47f5a58017c7acbb1c39 2023-03-27 04:58:31,257 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/fa67c5ca641f47f5a58017c7acbb1c39, entries=150, sequenceid=117, filesize=11.6 K 2023-03-27 04:58:31,258 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for a34144f135b39d6005ad383002fabe3a in 1004ms, sequenceid=117, compaction requested=true 2023-03-27 04:58:31,258 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:31,258 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:31,258 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:31,258 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:31,258 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,258 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:31,258 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:31,259 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:31,259 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:31,260 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:31,260 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:31,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:31,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:31,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@768ad420[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:31,263 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:31,263 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:31,267 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:31,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:31,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:31,267 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:31,267 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:31,267 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:31,267 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,268 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:31,269 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:31,269 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:31,269 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,269 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:31,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:31,272 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:31,272 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:31,272 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:31,272 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:31,272 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:31,273 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:31,273 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:31,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:31,273 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:31,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:31,274 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:31,276 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,278 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,278 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:31,278 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:31,278 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,278 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:31,278 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:31,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:31,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:31,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:31,278 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:31,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:31,278 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:31,278 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:31,278 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:31,278 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:31,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:31,278 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:31,378 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:31,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:31,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:31,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:31,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:31,381 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:31,381 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:31,381 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:31,381 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:31,381 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:31,381 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:31,382 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,382 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:31,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:31,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:31,382 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,383 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:31,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:31,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:31,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:31,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:31,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:31,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:31,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:31,384 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:31,384 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:31,384 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:31,384 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:31,384 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:31,384 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:31,384 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:31,384 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-27 04:58:31,384 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:31,384 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:31,384 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:31,384 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:31,384 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:31,384 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:31,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:31,559 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:31,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893171578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893171578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893171579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893171580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893171581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893171682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893171682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893171682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893171683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893171684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,811 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327ca38303d397340b7881b71ffa8194057 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327ca38303d397340b7881b71ffa8194057 2023-03-27 04:58:31,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893171885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893171886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893171886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893171886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:31,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:31,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893171888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:32,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893172188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:32,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893172189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:32,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893172191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:32,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893172191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:32,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893172192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,216 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=136, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/642d532ee37345bb8a64a3102af6af1d 2023-03-27 04:58:32,645 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/37a7d8a689cf4e2ea77e1e8168a84588 2023-03-27 04:58:32,667 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/535e96ea80404e45b9b0f00a8f040ab0 2023-03-27 04:58:32,671 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/642d532ee37345bb8a64a3102af6af1d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/642d532ee37345bb8a64a3102af6af1d 2023-03-27 04:58:32,674 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/642d532ee37345bb8a64a3102af6af1d, entries=150, sequenceid=136, filesize=25.3 K 2023-03-27 04:58:32,675 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/37a7d8a689cf4e2ea77e1e8168a84588 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/37a7d8a689cf4e2ea77e1e8168a84588 2023-03-27 04:58:32,677 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/37a7d8a689cf4e2ea77e1e8168a84588, entries=150, sequenceid=136, filesize=11.8 K 2023-03-27 04:58:32,678 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/535e96ea80404e45b9b0f00a8f040ab0 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/535e96ea80404e45b9b0f00a8f040ab0 2023-03-27 04:58:32,680 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/535e96ea80404e45b9b0f00a8f040ab0, entries=150, sequenceid=136, filesize=11.8 K 2023-03-27 04:58:32,681 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for a34144f135b39d6005ad383002fabe3a in 1297ms, sequenceid=136, compaction requested=true 2023-03-27 04:58:32,681 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:32,681 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:32,681 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:32,681 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:32,681 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,681 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:32,681 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:32,683 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,683 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:32,684 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,684 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:32,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:32,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:32,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:32,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:32,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11448057[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:32,685 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:32,685 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,685 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,686 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:32,686 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:32,686 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:32,686 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,686 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:32,686 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,686 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:32,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:32,686 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:32,687 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:32,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:32,688 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:32,688 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:32,689 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:32,689 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:32,689 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:32,694 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,694 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,694 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:32,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,695 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,695 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-27 04:58:32,695 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:32,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:32,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:32,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:32,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:32,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,696 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,698 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:32,698 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:32,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:32,698 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,698 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:32,698 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:32,698 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,698 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,698 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,698 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,698 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:32,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:32,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:32,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:32,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:32,699 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:32,699 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:32,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:32,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:32,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893172708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:32,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893172708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:32,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893172709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:32,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893172709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:32,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893172709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,799 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:32,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:32,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:32,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:32,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:32,802 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:32,802 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:32,802 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:32,802 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:32,802 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,802 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:32,803 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:32,803 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:32,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:32,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:32,804 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,804 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:32,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:32,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:32,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:32,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:32,805 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:32,805 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:32,805 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:32,806 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:32,806 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:32,806 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:32,806 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:32,806 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:32,806 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:32,806 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:32,806 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:32,806 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,806 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:32,806 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:32,807 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,807 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:32,808 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,808 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:32,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:32,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:32,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:32,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:32,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4574f82[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:32,809 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:32,809 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,810 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:32,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:32,810 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:32,811 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,811 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:32,811 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:32,811 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:32,811 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:32,812 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:32,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:32,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:32,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:32,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:32,815 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:32,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:32,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:32,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:32,815 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:32,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:32,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893172813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:32,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893172813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:32,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893172813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:32,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893172813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:32,816 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,816 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:32,816 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893172813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:32,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:32,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:32,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,818 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,819 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,821 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,821 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:32,821 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:32,821 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,821 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,821 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,821 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,821 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:32,821 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:32,821 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:32,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:32,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:32,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:32,821 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:32,821 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:32,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:32,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:32,921 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:32,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:32,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:32,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:32,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:32,925 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:32,925 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:32,925 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:32,925 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:32,925 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,925 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:32,926 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:32,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:32,926 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,926 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,926 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:32,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:32,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:32,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:32,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:32,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:32,928 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:32,928 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:32,928 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:32,928 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:32,928 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:32,928 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:32,928 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:32,928 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:32,928 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:32,928 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:32,929 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:32,929 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,929 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:32,929 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:32,929 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,929 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:32,930 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,930 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:32,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:32,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:32,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32b9b59f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:32,932 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:32,932 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,933 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,933 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:32,934 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:32,934 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:32,934 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,934 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:32,934 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:32,934 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:32,934 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:32,935 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:32,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:32,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:32,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:32,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:32,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:32,941 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:32,941 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:32,941 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:32,941 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:32,941 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:32,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:32,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:32,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:32,949 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:32,951 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:32,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:32,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:32,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:32,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:32,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,953 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:32,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:32,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:32,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:32,953 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:32,953 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:32,953 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:32,953 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:32,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:33,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:33,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:33,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893173017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893173017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:33,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893173017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:33,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893173018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:33,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893173018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,054 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:33,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:33,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:33,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:33,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:33,064 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:33,064 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:33,064 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:33,064 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:33,064 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,064 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,065 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:33,065 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:33,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:33,065 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,065 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:33,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:33,066 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:33,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:33,066 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:33,066 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:33,066 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:33,066 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:33,066 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:33,066 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:33,066 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:33,066 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:33,066 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:33,066 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:33,066 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:33,066 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,066 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:33,066 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:33,067 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,067 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,067 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,067 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:33,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12324c3c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:33,068 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:33,068 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,069 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,069 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:33,069 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,069 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:33,069 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:33,069 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:33,071 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:33,071 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,071 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:33,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,071 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:33,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:33,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:33,072 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:33,072 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:33,072 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:33,072 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:33,072 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:33,073 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,073 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,073 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,073 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:33,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,074 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:33,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:33,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:33,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:33,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:33,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:33,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,076 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:33,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:33,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:33,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,076 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:33,076 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,077 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:33,104 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303272eff4ce5b007422d93b5f8b323f1956e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303272eff4ce5b007422d93b5f8b323f1956e 2023-03-27 04:58:33,177 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:33,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:33,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:33,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:33,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:33,179 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:33,179 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:33,179 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:33,179 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:33,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,180 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:33,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:33,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:33,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,181 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:33,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:33,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:33,181 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:33,181 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:33,181 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:33,182 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:33,182 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:33,182 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:33,182 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:33,182 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:33,182 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:33,182 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:33,182 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:33,182 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,182 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:33,182 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:33,183 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,183 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,183 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,183 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:33,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27412015[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:33,184 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:33,184 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,184 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,184 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:33,185 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:33,185 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:33,185 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,185 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:33,185 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,185 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:33,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,185 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:33,185 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:33,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:33,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:33,187 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:33,187 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:33,187 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:33,187 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:33,187 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:33,188 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,188 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,188 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:33,188 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,189 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,190 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,190 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,190 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,190 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:33,191 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:33,191 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:33,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:33,191 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:33,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:33,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:33,191 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,191 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,191 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:33,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:33,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:33,191 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:33,191 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:33,291 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:33,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:33,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:33,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:33,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:33,294 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:33,294 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:33,294 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:33,294 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:33,294 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,294 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,295 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:33,295 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:33,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:33,295 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,295 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:33,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:33,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:33,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:33,296 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:33,296 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:33,296 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:33,296 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:33,297 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:33,297 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:33,297 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:33,297 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:33,297 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:33,297 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:33,297 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:33,297 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,297 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:33,297 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:33,298 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,298 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,298 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,298 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:33,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19544020[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:33,300 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:33,300 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,300 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,300 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:33,301 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:33,301 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:33,301 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,301 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:33,301 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:33,301 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:33,301 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,301 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:33,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:33,304 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:33,304 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:33,304 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:33,304 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:33,304 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:33,306 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,306 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,306 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:33,307 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,308 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,309 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:33,309 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:33,309 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,309 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:33,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:33,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:33,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:33,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:33,309 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,310 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,310 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,310 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,310 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,310 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:33,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:33,310 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:33,310 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:33,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:33,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:33,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893173321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:33,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893173322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:33,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893173322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:33,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893173323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:33,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893173323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,410 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:33,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:33,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:33,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:33,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:33,422 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:33,422 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:33,422 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:33,422 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:33,423 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,423 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,423 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,423 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:33,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:33,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:33,423 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,424 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:33,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:33,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:33,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:33,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:33,426 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:33,426 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:33,426 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:33,426 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:33,426 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:33,426 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:33,426 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:33,426 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:33,426 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:33,426 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:33,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:33,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:33,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:33,427 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,428 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,428 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:33,428 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:33,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66be0bd8[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:33,430 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:33,430 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,430 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,431 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:33,431 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:33,431 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:33,431 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,431 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:33,431 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:33,431 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:33,431 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:33,432 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:33,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:33,434 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:33,434 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:33,434 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:33,434 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:33,434 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:33,436 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,436 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,436 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:33,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,436 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,438 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,439 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,439 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,439 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,439 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:33,439 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:33,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:33,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:33,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:33,439 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:33,439 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:33,439 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:33,439 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,439 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,439 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:33,439 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:33,508 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=152, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/3b57dcd96b2b4dbfb3e35c5b42dc7bc3 2023-03-27 04:58:33,540 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:33,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:33,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:33,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:33,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:33,543 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:33,543 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:33,543 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:33,543 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:33,543 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,543 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,544 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:33,544 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:33,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:33,544 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,544 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:33,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:33,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:33,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:33,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:33,545 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:33,545 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:33,545 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:33,545 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:33,545 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:33,545 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:33,545 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:33,545 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:33,545 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:33,545 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:33,546 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:33,546 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,546 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:33,546 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:33,546 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,546 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,547 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,547 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:33,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:33,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66edf89f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:33,548 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:33,548 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,549 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,549 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:33,549 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,549 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:33,549 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:33,549 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:33,550 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,550 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:33,550 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:33,550 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:33,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:33,552 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:33,552 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:33,552 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:33,552 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:33,552 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:33,552 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,552 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,552 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:33,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:33,553 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,554 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,555 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:33,555 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:33,555 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:33,555 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:33,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:33,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:33,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:33,555 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,555 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,556 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,556 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,556 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,556 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,556 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:33,556 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:33,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:33,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:33,656 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:33,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:33,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:33,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:33,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:33,658 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:33,658 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:33,659 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:33,659 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:33,659 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,659 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,660 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,660 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:33,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:33,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:33,660 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,660 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:33,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:33,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:33,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:33,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:33,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:33,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:33,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:33,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:33,661 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:33,661 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:33,661 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:33,661 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:33,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:33,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:33,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:33,661 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:33,662 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,662 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,662 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,662 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:33,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:33,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48023ddf[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:33,663 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:33,663 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,664 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,664 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,664 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:33,664 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:33,664 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:33,664 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:33,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,665 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:33,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,665 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:33,665 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:33,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:33,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:33,666 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:33,667 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:33,667 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:33,667 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:33,667 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:33,667 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,667 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,667 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:33,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,668 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:33,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:33,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,669 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,670 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,670 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:33,670 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:33,670 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:33,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:33,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:33,670 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:33,670 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,671 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:33,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:33,671 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,671 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:33,671 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,671 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,671 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:33,771 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:33,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:33,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:33,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:33,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:33,773 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:33,773 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:33,773 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:33,773 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:33,774 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,774 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,774 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:33,774 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:33,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:33,775 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,775 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:33,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:33,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:33,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:33,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:33,775 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:33,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:33,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:33,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:33,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:33,776 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:33,776 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:33,776 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:33,776 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:33,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:33,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:33,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:33,776 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:33,777 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,777 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,777 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,777 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:33,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:33,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@736c31bf[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:33,778 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:33,778 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,779 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,779 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:33,779 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:33,779 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:33,779 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,779 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:33,779 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,779 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:33,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,779 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:33,780 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:33,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:33,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:33,781 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:33,781 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:33,781 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:33,781 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:33,781 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:33,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:33,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,782 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,782 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:33,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:33,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,783 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:33,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:33,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:33,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:33,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:33,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:33,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:33,786 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:33,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,786 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:33,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:33,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:33,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,786 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:33,786 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:33,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50916 deadline: 1679893173825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:33,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50974 deadline: 1679893173825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:33,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50896 deadline: 1679893173827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:33,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50936 deadline: 1679893173831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:33,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:50926 deadline: 1679893173831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,886 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:33,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:33,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:33,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:33,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:33,889 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:33,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:33,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:33,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:33,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,889 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,890 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,890 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:33,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:33,890 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,890 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:33,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:33,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:33,891 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:33,891 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:33,891 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:33,891 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:33,891 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:33,891 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:33,891 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:33,891 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:33,892 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:33,892 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:33,892 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:33,892 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,892 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:33,892 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:33,892 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,892 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,893 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,893 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:33,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:33,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@651a04c1[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:33,893 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:33,894 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,894 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,894 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:33,894 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,894 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:33,894 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:33,894 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:33,895 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:33,895 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:33,895 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,895 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:33,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:33,896 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:33,896 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:33,896 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:33,896 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:33,896 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:33,897 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,897 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:33,897 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:33,898 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,898 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:33,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:33,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:33,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:33,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:33,900 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:33,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:33,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:33,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:33,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:33,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:33,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:33,900 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:33,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:33,900 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:33,900 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:33,920 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/d71b50fb7eb74af79fa67aec53c79d3c 2023-03-27 04:58:34,000 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:34,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:34,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:34,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:34,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:34,003 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:34,003 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:34,003 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:34,003 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:34,003 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,003 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,004 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,004 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:34,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:34,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:34,004 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,004 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:34,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:34,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:34,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:34,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:34,005 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:34,005 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:34,005 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:34,005 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:34,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:34,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:34,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:34,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,006 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,006 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:34,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c3a21fd[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:34,008 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:34,008 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,008 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,009 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,009 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:34,009 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:34,009 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:34,009 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:34,010 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:34,010 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:34,010 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:34,010 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:34,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:34,012 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:34,012 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:34,012 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:34,012 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:34,012 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:34,013 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,013 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,013 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:34,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,013 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:34,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:34,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,014 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:34,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:34,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:34,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,017 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:34,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:34,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:34,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:34,018 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:34,018 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:34,018 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:34,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:34,118 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:34,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:34,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:34,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:34,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:34,122 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:34,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:34,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:34,122 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:34,123 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,123 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,123 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:34,123 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:34,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:34,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,124 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,125 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:34,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:34,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:34,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:34,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:34,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:34,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:34,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:34,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:34,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:34,126 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:34,126 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:34,126 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:34,126 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:34,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:34,126 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:34,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:34,127 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:34,128 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,128 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,128 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,129 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:34,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@350f0560[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:34,129 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:34,129 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,130 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,131 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:34,131 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:34,131 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:34,131 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,131 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:34,132 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:34,132 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:34,132 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,132 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:34,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:34,135 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:34,135 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:34,135 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:34,135 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:34,135 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:34,137 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,137 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:34,137 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,138 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,139 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,141 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:34,141 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:34,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:34,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:34,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:34,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:34,142 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,142 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,142 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,142 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:34,142 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:34,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:34,142 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,142 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,142 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,143 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:34,143 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:34,243 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:34,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:34,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:34,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:34,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:34,246 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:34,246 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:34,246 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:34,246 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:34,246 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,246 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,247 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:34,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:34,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:34,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,247 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:34,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:34,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:34,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:34,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:34,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:34,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:34,248 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:34,249 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:34,249 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:34,249 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:34,249 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:34,249 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:34,249 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:34,249 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:34,249 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:34,249 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,249 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:34,249 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:34,250 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,250 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,250 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,250 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:34,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@277c3bb7[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:34,251 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:34,252 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,252 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:34,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:34,252 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,252 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:34,252 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:34,253 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:34,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:34,253 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:34,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:34,255 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:34,255 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:34,255 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:34,255 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:34,255 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:34,256 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,256 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,256 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:34,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:34,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:34,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,257 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,258 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,259 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,259 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,259 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,259 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:34,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:34,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:34,259 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,259 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:34,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:34,259 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:34,259 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:34,259 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:34,259 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:34,260 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,260 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:34,328 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/b947a99f60e845e18d80aaf7afa63c58 2023-03-27 04:58:34,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/3b57dcd96b2b4dbfb3e35c5b42dc7bc3 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/3b57dcd96b2b4dbfb3e35c5b42dc7bc3 2023-03-27 04:58:34,334 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/3b57dcd96b2b4dbfb3e35c5b42dc7bc3, entries=150, sequenceid=152, filesize=25.3 K 2023-03-27 04:58:34,335 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/d71b50fb7eb74af79fa67aec53c79d3c as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/d71b50fb7eb74af79fa67aec53c79d3c 2023-03-27 04:58:34,337 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/d71b50fb7eb74af79fa67aec53c79d3c, entries=150, sequenceid=152, filesize=11.8 K 2023-03-27 04:58:34,338 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/b947a99f60e845e18d80aaf7afa63c58 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b947a99f60e845e18d80aaf7afa63c58 2023-03-27 04:58:34,342 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b947a99f60e845e18d80aaf7afa63c58, entries=150, sequenceid=152, filesize=11.8 K 2023-03-27 04:58:34,343 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for a34144f135b39d6005ad383002fabe3a in 1648ms, sequenceid=152, compaction requested=true 2023-03-27 04:58:34,343 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:34,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:34,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:34,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-27 04:58:34,344 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:58:34,343 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-27 04:58:34,347 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48050 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:58:34,347 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/B is initiating minor compaction (all files) 2023-03-27 04:58:34,347 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/B in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:34,347 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/b492350e1347453d94d85a33b0d0d5bd, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/7584a31f199846d39ce97150800eca50, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/37a7d8a689cf4e2ea77e1e8168a84588, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/d71b50fb7eb74af79fa67aec53c79d3c] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=46.9 K 2023-03-27 04:58:34,347 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting b492350e1347453d94d85a33b0d0d5bd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1679893109185 2023-03-27 04:58:34,348 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 7584a31f199846d39ce97150800eca50, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1679893109301 2023-03-27 04:58:34,349 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 168763 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-27 04:58:34,349 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/A is initiating minor compaction (all files) 2023-03-27 04:58:34,349 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/A in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:34,349 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/283e844b2d1c4cd99de1fd67592124cc, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/c1d2d6cd788c472591d611811df08ef7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/fcbe22d7477f490dba0efbe26e76f4d4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/86be74830c85442dbda54c9fde901971, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/642d532ee37345bb8a64a3102af6af1d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/3b57dcd96b2b4dbfb3e35c5b42dc7bc3] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=164.8 K 2023-03-27 04:58:34,349 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 37a7d8a689cf4e2ea77e1e8168a84588, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679893110432 2023-03-27 04:58:34,350 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 283e844b2d1c4cd99de1fd67592124cc, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679893107925 2023-03-27 04:58:34,350 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting d71b50fb7eb74af79fa67aec53c79d3c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1679893111578 2023-03-27 04:58:34,351 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting c1d2d6cd788c472591d611811df08ef7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679893108562 2023-03-27 04:58:34,351 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting fcbe22d7477f490dba0efbe26e76f4d4, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1679893109182 2023-03-27 04:58:34,352 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 86be74830c85442dbda54c9fde901971, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1679893109301 2023-03-27 04:58:34,352 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 642d532ee37345bb8a64a3102af6af1d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679893110432 2023-03-27 04:58:34,353 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 3b57dcd96b2b4dbfb3e35c5b42dc7bc3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1679893111578 2023-03-27 04:58:34,360 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:34,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:34,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:34,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:34,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:34,370 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:34,370 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:34,370 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:34,370 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:34,370 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,371 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,371 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#B#compaction#491 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:34,371 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:34,371 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:34,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:34,372 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,372 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:34,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:34,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:34,380 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:34,380 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:34,380 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:34,382 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:34,382 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:34,382 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:34,382 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:34,382 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-27 04:58:34,382 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:34,382 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:34,382 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:34,382 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:34,382 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:34,382 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:34,400 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#A#compaction#492 average throughput is 0.76 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:34,405 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/369ffe9420df42cba7a87022c882f6bf as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/369ffe9420df42cba7a87022c882f6bf 2023-03-27 04:58:34,408 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a34144f135b39d6005ad383002fabe3a/B of a34144f135b39d6005ad383002fabe3a into 369ffe9420df42cba7a87022c882f6bf(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:34,408 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:34,408 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/B, priority=12, startTime=1679893114343; duration=0sec 2023-03-27 04:58:34,408 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:34,408 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:58:34,409 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48050 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:58:34,409 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/C is initiating minor compaction (all files) 2023-03-27 04:58:34,409 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/C in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:34,409 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/c04df40d242e461ea799653acf1e7650, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/fa67c5ca641f47f5a58017c7acbb1c39, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/535e96ea80404e45b9b0f00a8f040ab0, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b947a99f60e845e18d80aaf7afa63c58] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=46.9 K 2023-03-27 04:58:34,409 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting c04df40d242e461ea799653acf1e7650, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1679893109185 2023-03-27 04:58:34,410 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting fa67c5ca641f47f5a58017c7acbb1c39, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1679893109301 2023-03-27 04:58:34,410 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 535e96ea80404e45b9b0f00a8f040ab0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679893110432 2023-03-27 04:58:34,410 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting b947a99f60e845e18d80aaf7afa63c58, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1679893111578 2023-03-27 04:58:34,417 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#C#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:34,421 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327fac0094cc0ad46eb9eaa0b8f346e7896 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327fac0094cc0ad46eb9eaa0b8f346e7896 2023-03-27 04:58:34,434 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=169, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/51fe687a6eae4ddbbaf2168dd462ebb5 2023-03-27 04:58:34,457 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/0df06ca66f8544a0927983c60bb35fb3 2023-03-27 04:58:34,469 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/83998e74701a49c99519f06c62636cf5 2023-03-27 04:58:34,473 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/51fe687a6eae4ddbbaf2168dd462ebb5 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/51fe687a6eae4ddbbaf2168dd462ebb5 2023-03-27 04:58:34,477 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/51fe687a6eae4ddbbaf2168dd462ebb5, entries=150, sequenceid=169, filesize=25.3 K 2023-03-27 04:58:34,478 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/0df06ca66f8544a0927983c60bb35fb3 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/0df06ca66f8544a0927983c60bb35fb3 2023-03-27 04:58:34,481 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/0df06ca66f8544a0927983c60bb35fb3, entries=150, sequenceid=169, filesize=11.8 K 2023-03-27 04:58:34,482 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/83998e74701a49c99519f06c62636cf5 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/83998e74701a49c99519f06c62636cf5 2023-03-27 04:58:34,486 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/83998e74701a49c99519f06c62636cf5, entries=150, sequenceid=169, filesize=11.8 K 2023-03-27 04:58:34,487 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for a34144f135b39d6005ad383002fabe3a in 105ms, sequenceid=169, compaction requested=false 2023-03-27 04:58:34,487 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:34,487 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:34,487 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:34,487 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:34,487 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,487 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:34,488 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:34,489 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,489 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,489 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,489 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:34,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f50f95[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:34,490 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:34,491 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,491 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,491 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:34,492 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:34,492 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:34,492 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,492 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:34,492 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,493 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:34,493 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:34,493 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:34,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:34,499 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:34,499 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:34,499 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:34,499 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:34,499 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:34,502 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,502 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,502 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:34,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,503 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:34,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:34,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,504 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,506 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:34,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:34,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:34,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:34,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:34,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:34,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:34,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:34,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,506 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:34,506 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,506 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:34,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:34,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:34,606 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:34,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:34,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:34,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:34,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:34,609 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:34,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:34,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:34,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:34,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,609 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,610 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,610 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:34,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:34,610 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,610 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,611 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:34,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:34,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:34,611 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:34,611 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:34,611 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:34,611 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:34,611 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:34,611 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:34,611 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:34,612 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:34,612 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:34,612 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:34,612 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:34,612 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,612 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:34,612 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:34,612 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,612 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,613 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,613 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:34,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f7de1ff[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:34,614 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:34,614 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,614 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,614 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,614 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:34,614 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:34,615 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:34,615 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:34,615 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,615 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:34,615 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:34,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,615 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:34,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:34,617 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:34,617 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:34,617 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:34,617 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:34,617 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:34,618 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,618 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,618 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:34,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,618 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:34,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:34,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,619 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,621 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:34,621 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:34,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:34,621 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,621 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,621 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:34,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:34,621 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,621 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,621 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,621 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,621 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:34,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:34,621 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:34,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:34,621 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:34,721 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:34,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:34,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:34,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:34,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:34,724 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:34,725 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:34,725 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:34,725 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:34,725 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,725 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,726 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,726 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:34,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:34,726 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,726 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:34,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:34,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:34,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:34,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:34,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:34,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:34,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:34,727 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:34,727 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:34,727 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:34,727 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:34,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:34,727 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:34,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:34,728 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:34,729 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,729 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,730 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,730 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:34,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6294c575[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:34,736 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:34,736 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,740 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,740 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,740 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:34,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:34,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:34,740 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:34,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:34,741 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,741 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:34,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:34,745 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:34,745 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:34,745 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:34,745 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:34,745 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:34,746 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,747 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:34,747 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,747 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:34,750 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,752 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:34,752 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:34,752 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:34,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:34,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:34,752 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,752 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:34,752 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:34,752 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,752 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:34,752 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:34,752 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,753 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:34,753 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,834 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/20d6c2e543294ba3bd9c0ebe38fcd4a6 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/20d6c2e543294ba3bd9c0ebe38fcd4a6 2023-03-27 04:58:34,853 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:34,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:34,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:34,858 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a34144f135b39d6005ad383002fabe3a/C of a34144f135b39d6005ad383002fabe3a into 20d6c2e543294ba3bd9c0ebe38fcd4a6(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:34,858 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:34,858 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/C, priority=12, startTime=1679893114343; duration=0sec 2023-03-27 04:58:34,858 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:34,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:34,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:34,859 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:34,859 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:34,859 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:34,859 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:34,859 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,859 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,860 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:34,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:34,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:34,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:34,862 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:34,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,862 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:34,862 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:34,862 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:34,862 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:34,862 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:34,862 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:34,862 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-03-27 04:58:34,863 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:34,863 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:34,863 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:34,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:34,863 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:34,863 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:34,865 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:34,866 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:34,883 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327e9d99a1351264a2a8006c4bdafa93c2d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e9d99a1351264a2a8006c4bdafa93c2d 2023-03-27 04:58:34,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:34,887 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:34,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679893114895, max timestamp=1679893114932Num uniques 859; 2023-03-27 04:58:34,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:58:34,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679893114895, max timestamp=1679893114932Num uniques 859; 2023-03-27 04:58:34,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679893114895, max timestamp=1679893114932Num uniques 859; 2023-03-27 04:58:34,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:58:34,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:58:34,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679893114950, max timestamp=1679893114957Num uniques 1068; 2023-03-27 04:58:34,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679893114950, max timestamp=1679893114957Num uniques 1068; 2023-03-27 04:58:34,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-27 04:58:34,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679893114950, max timestamp=1679893114957Num uniques 1068; 2023-03-27 04:58:34,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-27 04:58:34,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-27 04:58:34,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679893114967, max timestamp=1679893114974Num uniques 1127; 2023-03-27 04:58:34,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-03-27 04:58:34,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679893114967, max timestamp=1679893114974Num uniques 1127; 2023-03-27 04:58:34,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-03-27 04:58:34,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679893114967, max timestamp=1679893114974Num uniques 1027; 2023-03-27 04:58:34,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-03-27 04:58:34,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1679893114984, max timestamp=1679893114992Num uniques 1136; 2023-03-27 04:58:34,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1679893114984, max timestamp=1679893114992Num uniques 1136; 2023-03-27 04:58:34,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1679893114984, max timestamp=1679893114992Num uniques 1136; 2023-03-27 04:58:34,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=15386, suffix heap size=146112,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=336 2023-03-27 04:58:34,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=15386, suffix heap size=146112,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=336 2023-03-27 04:58:34,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=15386, suffix heap size=146112,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=336 2023-03-27 04:58:35,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1679893115000, max timestamp=1679893115007Num uniques 1195; 2023-03-27 04:58:35,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1679893115000, max timestamp=1679893115007Num uniques 1195; 2023-03-27 04:58:35,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=18090, suffix heap size=152248,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=395 2023-03-27 04:58:35,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=18090, suffix heap size=152248,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=395 2023-03-27 04:58:35,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1679893115000, max timestamp=1679893115007Num uniques 1195; 2023-03-27 04:58:35,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=18090, suffix heap size=152248,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=395 2023-03-27 04:58:35,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679893115012, max timestamp=1679893115027Num uniques 1254; 2023-03-27 04:58:35,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=57430, new segment data size=9340, suffix heap size=158384,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=204 2023-03-27 04:58:35,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679893115012, max timestamp=1679893115027Num uniques 1254; 2023-03-27 04:58:35,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=57430, new segment data size=9340, suffix heap size=158384,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=204 2023-03-27 04:58:35,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679893115012, max timestamp=1679893115027Num uniques 1254; 2023-03-27 04:58:35,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=57430, new segment data size=9340, suffix heap size=158384,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=204 2023-03-27 04:58:35,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679893115042, max timestamp=1679893115047Num uniques 1063; 2023-03-27 04:58:35,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-03-27 04:58:35,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679893115042, max timestamp=1679893115047Num uniques 1063; 2023-03-27 04:58:35,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-03-27 04:58:35,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679893115042, max timestamp=1679893115047Num uniques 1063; 2023-03-27 04:58:35,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-03-27 04:58:35,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679893115054, max timestamp=1679893115063Num uniques 1022; 2023-03-27 04:58:35,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-27 04:58:35,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679893115054, max timestamp=1679893115063Num uniques 1022; 2023-03-27 04:58:35,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-27 04:58:35,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679893115054, max timestamp=1679893115063Num uniques 1022; 2023-03-27 04:58:35,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=6870, suffix heap size=134256,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=150 2023-03-27 04:58:35,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679893115071, max timestamp=1679893115078Num uniques 1081; 2023-03-27 04:58:35,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-03-27 04:58:35,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679893115071, max timestamp=1679893115078Num uniques 1009; 2023-03-27 04:58:35,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679893115071, max timestamp=1679893115078Num uniques 1081; 2023-03-27 04:58:35,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=12866, suffix heap size=132904,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=281 2023-03-27 04:58:35,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-03-27 04:58:35,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679893115096, max timestamp=1679893115099Num uniques 1140; 2023-03-27 04:58:35,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679893115096, max timestamp=1679893115099Num uniques 1140; 2023-03-27 04:58:35,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679893115096, max timestamp=1679893115099Num uniques 1140; 2023-03-27 04:58:35,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-03-27 04:58:35,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-03-27 04:58:35,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-03-27 04:58:35,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9104, totalHeapSize=20936, min timestamp=1679893115107, max timestamp=1679893115116Num uniques 1049; 2023-03-27 04:58:35,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48034, new segment data size=9104, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-27 04:58:35,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9104, totalHeapSize=20936, min timestamp=1679893115107, max timestamp=1679893115116Num uniques 1049; 2023-03-27 04:58:35,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48034, new segment data size=9104, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-27 04:58:35,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9104, totalHeapSize=20936, min timestamp=1679893115107, max timestamp=1679893115116Num uniques 1049; 2023-03-27 04:58:35,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48034, new segment data size=9104, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-27 04:58:35,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1679893115129, max timestamp=1679893115133Num uniques 1058; 2023-03-27 04:58:35,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1679893115129, max timestamp=1679893115133Num uniques 1058; 2023-03-27 04:58:35,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-03-27 04:58:35,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1679893115129, max timestamp=1679893115133Num uniques 1058; 2023-03-27 04:58:35,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-03-27 04:58:35,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-03-27 04:58:35,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679893115146, max timestamp=1679893115149Num uniques 1117; 2023-03-27 04:58:35,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-03-27 04:58:35,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679893115146, max timestamp=1679893115149Num uniques 1117; 2023-03-27 04:58:35,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=6870, suffix heap size=144136,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=150 2023-03-27 04:58:35,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679893115146, max timestamp=1679893115149Num uniques 1117; 2023-03-27 04:58:35,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-03-27 04:58:35,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1679893115152, max timestamp=1679893115164Num uniques 1009; 2023-03-27 04:58:35,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10346, suffix heap size=132904,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=226 2023-03-27 04:58:35,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1679893115152, max timestamp=1679893115164Num uniques 1026; 2023-03-27 04:58:35,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-03-27 04:58:35,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1679893115152, max timestamp=1679893115164Num uniques 1026; 2023-03-27 04:58:35,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-03-27 04:58:35,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1679893115174, max timestamp=1679893115179Num uniques 1085; 2023-03-27 04:58:35,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=13050, suffix heap size=140808,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=285 2023-03-27 04:58:35,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1679893115174, max timestamp=1679893115179Num uniques 1085; 2023-03-27 04:58:35,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=13050, suffix heap size=140808,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=285 2023-03-27 04:58:35,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1679893115174, max timestamp=1679893115179Num uniques 1085; 2023-03-27 04:58:35,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-03-27 04:58:35,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11164, totalHeapSize=25616, min timestamp=1679893115210, max timestamp=1679893115219Num uniques 1144; 2023-03-27 04:58:35,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=52384, new segment data size=11164, suffix heap size=146944,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=244 2023-03-27 04:58:35,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11164, totalHeapSize=25616, min timestamp=1679893115210, max timestamp=1679893115219Num uniques 1144; 2023-03-27 04:58:35,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11164, totalHeapSize=25616, min timestamp=1679893115210, max timestamp=1679893115219Num uniques 1044; 2023-03-27 04:58:35,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52384, new segment data size=11164, suffix heap size=146944,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=244 2023-03-27 04:58:35,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47804, new segment data size=11164, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-03-27 04:58:35,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679893115241, max timestamp=1679893115243Num uniques 1103; 2023-03-27 04:58:35,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-03-27 04:58:35,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679893115241, max timestamp=1679893115242Num uniques 1103; 2023-03-27 04:58:35,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-03-27 04:58:35,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679893115241, max timestamp=1679893115243Num uniques 1103; 2023-03-27 04:58:35,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-03-27 04:58:35,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1679893115253, max timestamp=1679893115258Num uniques 1012; 2023-03-27 04:58:35,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1679893115253, max timestamp=1679893115258Num uniques 1012; 2023-03-27 04:58:35,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-03-27 04:58:35,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1679893115253, max timestamp=1679893115258Num uniques 1012; 2023-03-27 04:58:35,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-03-27 04:58:35,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-03-27 04:58:35,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1679893115267, max timestamp=1679893115272Num uniques 1121; 2023-03-27 04:58:35,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=12406, suffix heap size=144552,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=271 2023-03-27 04:58:35,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1679893115267, max timestamp=1679893115272Num uniques 1121; 2023-03-27 04:58:35,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=12406, suffix heap size=144552,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=271 2023-03-27 04:58:35,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1679893115267, max timestamp=1679893115272Num uniques 1121; 2023-03-27 04:58:35,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=12406, suffix heap size=144552,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=271 2023-03-27 04:58:35,287 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=177, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/e56de8e34969424bbdb26538d85fdb38 2023-03-27 04:58:35,287 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1679893115283, max timestamp=1679893115286Num uniques 1130; 2023-03-27 04:58:35,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=8240, suffix heap size=145488,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=180 2023-03-27 04:58:35,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1679893115283, max timestamp=1679893115286Num uniques 1130; 2023-03-27 04:58:35,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=8240, suffix heap size=145488,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=180 2023-03-27 04:58:35,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1679893115283, max timestamp=1679893115286Num uniques 1130; 2023-03-27 04:58:35,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=8240, suffix heap size=145488,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=180 2023-03-27 04:58:35,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10938, totalHeapSize=25096, min timestamp=1679893115303, max timestamp=1679893115311Num uniques 1039; 2023-03-27 04:58:35,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47578, new segment data size=10938, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-03-27 04:58:35,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10934, totalHeapSize=25096, min timestamp=1679893115303, max timestamp=1679893115311Num uniques 1039; 2023-03-27 04:58:35,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47574, new segment data size=10934, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-03-27 04:58:35,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10934, totalHeapSize=25096, min timestamp=1679893115303, max timestamp=1679893115311Num uniques 1039; 2023-03-27 04:58:35,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47574, new segment data size=10934, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-03-27 04:58:35,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9048, totalHeapSize=20832, min timestamp=1679893115328, max timestamp=1679893115334Num uniques 1098; 2023-03-27 04:58:35,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9058, totalHeapSize=20832, min timestamp=1679893115328, max timestamp=1679893115334Num uniques 1098; 2023-03-27 04:58:35,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50268, new segment data size=9048, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-03-27 04:58:35,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50278, new segment data size=9058, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-03-27 04:58:35,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9057, totalHeapSize=20832, min timestamp=1679893115328, max timestamp=1679893115334Num uniques 1098; 2023-03-27 04:58:35,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50277, new segment data size=9057, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-03-27 04:58:35,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1679893115350, max timestamp=1679893115355Num uniques 1057; 2023-03-27 04:58:35,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=7185, suffix heap size=137896,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=157 2023-03-27 04:58:35,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1679893115350, max timestamp=1679893115355Num uniques 1057; 2023-03-27 04:58:35,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=7185, suffix heap size=137896,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=157 2023-03-27 04:58:35,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1679893115350, max timestamp=1679893115355Num uniques 1057; 2023-03-27 04:58:35,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=7185, suffix heap size=137896,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=157 2023-03-27 04:58:35,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=316, cellSize=14466, totalHeapSize=33104, min timestamp=1679893115363, max timestamp=1679893115371Num uniques 1016; 2023-03-27 04:58:35,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=316, cellSize=14466, totalHeapSize=33104, min timestamp=1679893115363, max timestamp=1679893115371Num uniques 1016; 2023-03-27 04:58:35,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=14466, suffix heap size=133632,new segment heap size=33104 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=316 2023-03-27 04:58:35,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=14466, suffix heap size=133632,new segment heap size=33104 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=316 2023-03-27 04:58:35,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=316, cellSize=14466, totalHeapSize=33104, min timestamp=1679893115363, max timestamp=1679893115371Num uniques 1016; 2023-03-27 04:58:35,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=14466, suffix heap size=133632,new segment heap size=33104 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=316 2023-03-27 04:58:35,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1679893115379, max timestamp=1679893115384Num uniques 1175; 2023-03-27 04:58:35,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=53810, new segment data size=12590, suffix heap size=150168,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1175, new segment cells count=275 2023-03-27 04:58:35,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1679893115379, max timestamp=1679893115384Num uniques 1175; 2023-03-27 04:58:35,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=53810, new segment data size=12590, suffix heap size=150168,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1175, new segment cells count=275 2023-03-27 04:58:35,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1679893115379, max timestamp=1679893115384Num uniques 1175; 2023-03-27 04:58:35,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=53810, new segment data size=12590, suffix heap size=150168,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1175, new segment cells count=275 2023-03-27 04:58:35,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8414, totalHeapSize=19376, min timestamp=1679893115395, max timestamp=1679893115400Num uniques 1134; 2023-03-27 04:58:35,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51924, new segment data size=8414, suffix heap size=145904,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=184 2023-03-27 04:58:35,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8414, totalHeapSize=19376, min timestamp=1679893115395, max timestamp=1679893115400Num uniques 1134; 2023-03-27 04:58:35,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51924, new segment data size=8414, suffix heap size=145904,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=184 2023-03-27 04:58:35,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1679893115395, max timestamp=1679893115400Num uniques 1134; 2023-03-27 04:58:35,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51934, new segment data size=8424, suffix heap size=145904,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=184 2023-03-27 04:58:35,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1679893115409, max timestamp=1679893115414Num uniques 1043; 2023-03-27 04:58:35,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11118, totalHeapSize=25512, min timestamp=1679893115409, max timestamp=1679893115414Num uniques 1043; 2023-03-27 04:58:35,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47768, new segment data size=11128, suffix heap size=136440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=243 2023-03-27 04:58:35,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47758, new segment data size=11118, suffix heap size=136440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=243 2023-03-27 04:58:35,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11118, totalHeapSize=25512, min timestamp=1679893115409, max timestamp=1679893115414Num uniques 1043; 2023-03-27 04:58:35,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47758, new segment data size=11118, suffix heap size=136440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=243 2023-03-27 04:58:35,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9242, totalHeapSize=21248, min timestamp=1679893115420, max timestamp=1679893115433Num uniques 1102; 2023-03-27 04:58:35,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50462, new segment data size=9242, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-03-27 04:58:35,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9242, totalHeapSize=21248, min timestamp=1679893115420, max timestamp=1679893115433Num uniques 1102; 2023-03-27 04:58:35,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50462, new segment data size=9242, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-03-27 04:58:35,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9242, totalHeapSize=21248, min timestamp=1679893115420, max timestamp=1679893115433Num uniques 1102; 2023-03-27 04:58:35,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50462, new segment data size=9242, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-03-27 04:58:35,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1679893115444, max timestamp=1679893115449Num uniques 1061; 2023-03-27 04:58:35,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7365, totalHeapSize=16984, min timestamp=1679893115444, max timestamp=1679893115449Num uniques 1061; 2023-03-27 04:58:35,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1679893115444, max timestamp=1679893115449Num uniques 1061; 2023-03-27 04:58:35,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48585, new segment data size=7365, suffix heap size=138312,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=161 2023-03-27 04:58:35,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=7366, suffix heap size=138312,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=161 2023-03-27 04:58:35,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=7366, suffix heap size=138312,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=161 2023-03-27 04:58:35,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1679893115459, max timestamp=1679893115463Num uniques 1020; 2023-03-27 04:58:35,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=10070, suffix heap size=134048,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=220 2023-03-27 04:58:35,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1679893115459, max timestamp=1679893115463Num uniques 1020; 2023-03-27 04:58:35,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1679893115459, max timestamp=1679893115463Num uniques 1020; 2023-03-27 04:58:35,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=10070, suffix heap size=134048,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=220 2023-03-27 04:58:35,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=10070, suffix heap size=134048,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=220 2023-03-27 04:58:35,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1679893115466, max timestamp=1679893115504Num uniques 1079; 2023-03-27 04:58:35,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1679893115466, max timestamp=1679893115504Num uniques 1079; 2023-03-27 04:58:35,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=8194, suffix heap size=140184,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=179 2023-03-27 04:58:35,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=8194, suffix heap size=140184,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=179 2023-03-27 04:58:35,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1679893115466, max timestamp=1679893115504Num uniques 1079; 2023-03-27 04:58:35,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=8194, suffix heap size=140184,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=179 2023-03-27 04:58:35,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=288, cellSize=13188, totalHeapSize=30192, min timestamp=1679893115512, max timestamp=1679893115520Num uniques 1038; 2023-03-27 04:58:35,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=13188, suffix heap size=135920,new segment heap size=30192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=288 2023-03-27 04:58:35,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=288, cellSize=13188, totalHeapSize=30192, min timestamp=1679893115512, max timestamp=1679893115520Num uniques 1038; 2023-03-27 04:58:35,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=13188, suffix heap size=135920,new segment heap size=30192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=288 2023-03-27 04:58:35,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=288, cellSize=13188, totalHeapSize=30192, min timestamp=1679893115512, max timestamp=1679893115520Num uniques 1038; 2023-03-27 04:58:35,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=13188, suffix heap size=135920,new segment heap size=30192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=288 2023-03-27 04:58:35,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=297, cellSize=13602, totalHeapSize=31128, min timestamp=1679893115529, max timestamp=1679893115535Num uniques 1147; 2023-03-27 04:58:35,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=297, cellSize=13602, totalHeapSize=31128, min timestamp=1679893115529, max timestamp=1679893115535Num uniques 1147; 2023-03-27 04:58:35,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=52532, new segment data size=13602, suffix heap size=147256,new segment heap size=31128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1147, new segment cells count=297 2023-03-27 04:58:35,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=297, cellSize=13602, totalHeapSize=31128, min timestamp=1679893115529, max timestamp=1679893115535Num uniques 1147; 2023-03-27 04:58:35,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52532, new segment data size=13602, suffix heap size=147256,new segment heap size=31128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1147, new segment cells count=297 2023-03-27 04:58:35,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=52532, new segment data size=13602, suffix heap size=147256,new segment heap size=31128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1147, new segment cells count=297 2023-03-27 04:58:35,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=356, cellSize=16300, totalHeapSize=37264, min timestamp=1679893115546, max timestamp=1679893115553Num uniques 1156; 2023-03-27 04:58:35,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=52940, new segment data size=16300, suffix heap size=148192,new segment heap size=37264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1156, new segment cells count=356 2023-03-27 04:58:35,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=356, cellSize=16300, totalHeapSize=37264, min timestamp=1679893115546, max timestamp=1679893115553Num uniques 1156; 2023-03-27 04:58:35,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=356, cellSize=16300, totalHeapSize=37264, min timestamp=1679893115546, max timestamp=1679893115553Num uniques 1156; 2023-03-27 04:58:35,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=52940, new segment data size=16300, suffix heap size=148192,new segment heap size=37264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1156, new segment cells count=356 2023-03-27 04:58:35,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=52940, new segment data size=16300, suffix heap size=148192,new segment heap size=37264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1156, new segment cells count=356 2023-03-27 04:58:35,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=265, cellSize=12130, totalHeapSize=27800, min timestamp=1679893115565, max timestamp=1679893115570Num uniques 1215; 2023-03-27 04:58:35,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=55640, new segment data size=12130, suffix heap size=154328,new segment heap size=27800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1215, new segment cells count=265 2023-03-27 04:58:35,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=265, cellSize=12130, totalHeapSize=27800, min timestamp=1679893115565, max timestamp=1679893115570Num uniques 1215; 2023-03-27 04:58:35,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=265, cellSize=12130, totalHeapSize=27800, min timestamp=1679893115565, max timestamp=1679893115570Num uniques 1215; 2023-03-27 04:58:35,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=55640, new segment data size=12130, suffix heap size=154328,new segment heap size=27800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1215, new segment cells count=265 2023-03-27 04:58:35,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=55640, new segment data size=12130, suffix heap size=154328,new segment heap size=27800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1215, new segment cells count=265 2023-03-27 04:58:35,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10247, totalHeapSize=23536, min timestamp=1679893115583, max timestamp=1679893115591Num uniques 1124; 2023-03-27 04:58:35,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10244, totalHeapSize=23536, min timestamp=1679893115583, max timestamp=1679893115591Num uniques 1124; 2023-03-27 04:58:35,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51467, new segment data size=10247, suffix heap size=144864,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1124, new segment cells count=224 2023-03-27 04:58:35,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51464, new segment data size=10244, suffix heap size=144864,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1124, new segment cells count=224 2023-03-27 04:58:35,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10245, totalHeapSize=23536, min timestamp=1679893115583, max timestamp=1679893115591Num uniques 1124; 2023-03-27 04:58:35,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=51465, new segment data size=10245, suffix heap size=144864,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1124, new segment cells count=224 2023-03-27 04:58:35,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=283, cellSize=12958, totalHeapSize=29672, min timestamp=1679893115601, max timestamp=1679893115604Num uniques 1083; 2023-03-27 04:58:35,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=12958, suffix heap size=140600,new segment heap size=29672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=283 2023-03-27 04:58:35,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=283, cellSize=12958, totalHeapSize=29672, min timestamp=1679893115601, max timestamp=1679893115604Num uniques 1083; 2023-03-27 04:58:35,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=12958, suffix heap size=140600,new segment heap size=29672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=283 2023-03-27 04:58:35,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=283, cellSize=12958, totalHeapSize=29672, min timestamp=1679893115601, max timestamp=1679893115604Num uniques 1083; 2023-03-27 04:58:35,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=12958, suffix heap size=140600,new segment heap size=29672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=283 2023-03-27 04:58:35,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=192, cellSize=8792, totalHeapSize=20208, min timestamp=1679893115616, max timestamp=1679893115620Num uniques 1142; 2023-03-27 04:58:35,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=52302, new segment data size=8792, suffix heap size=146736,new segment heap size=20208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1142, new segment cells count=192 2023-03-27 04:58:35,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=192, cellSize=8792, totalHeapSize=20208, min timestamp=1679893115616, max timestamp=1679893115620Num uniques 1142; 2023-03-27 04:58:35,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=52302, new segment data size=8792, suffix heap size=146736,new segment heap size=20208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1142, new segment cells count=192 2023-03-27 04:58:35,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=192, cellSize=8788, totalHeapSize=20208, min timestamp=1679893115616, max timestamp=1679893115620Num uniques 1142; 2023-03-27 04:58:35,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=52298, new segment data size=8788, suffix heap size=146736,new segment heap size=20208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1142, new segment cells count=192 2023-03-27 04:58:35,634 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/c798d48c7dcf4d55a90620c4cf27ee1e as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/c798d48c7dcf4d55a90620c4cf27ee1e 2023-03-27 04:58:35,638 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a34144f135b39d6005ad383002fabe3a/A of a34144f135b39d6005ad383002fabe3a into c798d48c7dcf4d55a90620c4cf27ee1e(size=25.6 K), total size for store is 50.9 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-27 04:58:35,638 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:35,638 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/A, priority=10, startTime=1679893114343; duration=1sec 2023-03-27 04:58:35,638 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:35,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1679893115630, max timestamp=1679893115636Num uniques 1051; 2023-03-27 04:58:35,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48135, new segment data size=6915, suffix heap size=137272,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1051, new segment cells count=151 2023-03-27 04:58:35,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1679893115630, max timestamp=1679893115636Num uniques 1051; 2023-03-27 04:58:35,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48135, new segment data size=6915, suffix heap size=137272,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1051, new segment cells count=151 2023-03-27 04:58:35,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679893115630, max timestamp=1679893115636Num uniques 1051; 2023-03-27 04:58:35,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48126, new segment data size=7688, suffix heap size=137272,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1051, new segment cells count=168 2023-03-27 04:58:35,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=160, cellSize=7320, totalHeapSize=16880, min timestamp=1679893115641, max timestamp=1679893115649Num uniques 1027; 2023-03-27 04:58:35,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=160, cellSize=7320, totalHeapSize=16880, min timestamp=1679893115641, max timestamp=1679893115649Num uniques 1010; 2023-03-27 04:58:35,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=7320, suffix heap size=134776,new segment heap size=16880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=160 2023-03-27 04:58:35,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46250, new segment data size=7320, suffix heap size=133008,new segment heap size=16880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1010, new segment cells count=160 2023-03-27 04:58:35,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=160, cellSize=7320, totalHeapSize=16880, min timestamp=1679893115641, max timestamp=1679893115649Num uniques 1010; 2023-03-27 04:58:35,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46250, new segment data size=7320, suffix heap size=133008,new segment heap size=16880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1010, new segment cells count=160 2023-03-27 04:58:35,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=269, cellSize=12314, totalHeapSize=28216, min timestamp=1679893115659, max timestamp=1679893115664Num uniques 1019; 2023-03-27 04:58:35,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46664, new segment data size=12314, suffix heap size=133944,new segment heap size=28216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1019, new segment cells count=269 2023-03-27 04:58:35,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=269, cellSize=12314, totalHeapSize=28216, min timestamp=1679893115659, max timestamp=1679893115664Num uniques 1019; 2023-03-27 04:58:35,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46664, new segment data size=12314, suffix heap size=133944,new segment heap size=28216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1019, new segment cells count=269 2023-03-27 04:58:35,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=269, cellSize=12314, totalHeapSize=28216, min timestamp=1679893115659, max timestamp=1679893115664Num uniques 1019; 2023-03-27 04:58:35,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46664, new segment data size=12314, suffix heap size=133944,new segment heap size=28216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1019, new segment cells count=269 2023-03-27 04:58:35,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1679893115671, max timestamp=1679893115676Num uniques 1128; 2023-03-27 04:58:35,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51658, new segment data size=10438, suffix heap size=145280,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1128, new segment cells count=228 2023-03-27 04:58:35,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1679893115671, max timestamp=1679893115676Num uniques 1128; 2023-03-27 04:58:35,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51658, new segment data size=10438, suffix heap size=145280,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1128, new segment cells count=228 2023-03-27 04:58:35,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1679893115671, max timestamp=1679893115676Num uniques 1128; 2023-03-27 04:58:35,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51658, new segment data size=10438, suffix heap size=145280,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1128, new segment cells count=228 2023-03-27 04:58:35,697 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/1cbf8b2767b7414d9cfc13a612c08263 2023-03-27 04:58:35,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=187, cellSize=8545, totalHeapSize=19688, min timestamp=1679893115685, max timestamp=1679893115698Num uniques 1087; 2023-03-27 04:58:35,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=187, cellSize=8552, totalHeapSize=19688, min timestamp=1679893115685, max timestamp=1679893115698Num uniques 1087; 2023-03-27 04:58:35,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=187, cellSize=8542, totalHeapSize=19688, min timestamp=1679893115685, max timestamp=1679893115698Num uniques 1087; 2023-03-27 04:58:35,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49762, new segment data size=8542, suffix heap size=141016,new segment heap size=19688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=187 2023-03-27 04:58:35,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49772, new segment data size=8552, suffix heap size=141016,new segment heap size=19688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=187 2023-03-27 04:58:35,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49765, new segment data size=8545, suffix heap size=141016,new segment heap size=19688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=187 2023-03-27 04:58:35,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=296, cellSize=13556, totalHeapSize=31024, min timestamp=1679893115720, max timestamp=1679893115724Num uniques 1046; 2023-03-27 04:58:35,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47906, new segment data size=13556, suffix heap size=136752,new segment heap size=31024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1046, new segment cells count=296 2023-03-27 04:58:35,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=296, cellSize=13556, totalHeapSize=31024, min timestamp=1679893115720, max timestamp=1679893115724Num uniques 1046; 2023-03-27 04:58:35,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47906, new segment data size=13556, suffix heap size=136752,new segment heap size=31024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1046, new segment cells count=296 2023-03-27 04:58:35,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=296, cellSize=13556, totalHeapSize=31024, min timestamp=1679893115720, max timestamp=1679893115724Num uniques 1046; 2023-03-27 04:58:35,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47906, new segment data size=13556, suffix heap size=136752,new segment heap size=31024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1046, new segment cells count=296 2023-03-27 04:58:35,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=155, cellSize=7095, totalHeapSize=16360, min timestamp=1679893115736, max timestamp=1679893115742Num uniques 1155; 2023-03-27 04:58:35,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=52895, new segment data size=7095, suffix heap size=148088,new segment heap size=16360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1155, new segment cells count=155 2023-03-27 04:58:35,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=155, cellSize=7095, totalHeapSize=16360, min timestamp=1679893115736, max timestamp=1679893115742Num uniques 1155; 2023-03-27 04:58:35,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=52895, new segment data size=7095, suffix heap size=148088,new segment heap size=16360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1155, new segment cells count=155 2023-03-27 04:58:35,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=155, cellSize=7095, totalHeapSize=16360, min timestamp=1679893115736, max timestamp=1679893115742Num uniques 1155; 2023-03-27 04:58:35,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=52895, new segment data size=7095, suffix heap size=148088,new segment heap size=16360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1155, new segment cells count=155 2023-03-27 04:58:35,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=264, cellSize=12084, totalHeapSize=27696, min timestamp=1679893115750, max timestamp=1679893115755Num uniques 1014; 2023-03-27 04:58:35,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46434, new segment data size=12084, suffix heap size=133424,new segment heap size=27696 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1014, new segment cells count=264 2023-03-27 04:58:35,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=264, cellSize=12084, totalHeapSize=27696, min timestamp=1679893115750, max timestamp=1679893115755Num uniques 1014; 2023-03-27 04:58:35,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46434, new segment data size=12084, suffix heap size=133424,new segment heap size=27696 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1014, new segment cells count=264 2023-03-27 04:58:35,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=264, cellSize=12084, totalHeapSize=27696, min timestamp=1679893115750, max timestamp=1679893115755Num uniques 1014; 2023-03-27 04:58:35,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46434, new segment data size=12084, suffix heap size=133424,new segment heap size=27696 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1014, new segment cells count=264 2023-03-27 04:58:35,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=223, cellSize=10208, totalHeapSize=23432, min timestamp=1679893115763, max timestamp=1679893115767Num uniques 1123; 2023-03-27 04:58:35,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=223, cellSize=10208, totalHeapSize=23432, min timestamp=1679893115763, max timestamp=1679893115767Num uniques 1123; 2023-03-27 04:58:35,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51428, new segment data size=10208, suffix heap size=144760,new segment heap size=23432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1123, new segment cells count=223 2023-03-27 04:58:35,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=51428, new segment data size=10208, suffix heap size=144760,new segment heap size=23432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1123, new segment cells count=223 2023-03-27 04:58:35,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=223, cellSize=10208, totalHeapSize=23432, min timestamp=1679893115763, max timestamp=1679893115767Num uniques 1123; 2023-03-27 04:58:35,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=51428, new segment data size=10208, suffix heap size=144760,new segment heap size=23432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1123, new segment cells count=223 2023-03-27 04:58:35,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=282, cellSize=12912, totalHeapSize=29568, min timestamp=1679893115774, max timestamp=1679893115780Num uniques 1082; 2023-03-27 04:58:35,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49552, new segment data size=12912, suffix heap size=140496,new segment heap size=29568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1082, new segment cells count=282 2023-03-27 04:58:35,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=282, cellSize=12912, totalHeapSize=29568, min timestamp=1679893115774, max timestamp=1679893115780Num uniques 1082; 2023-03-27 04:58:35,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49552, new segment data size=12912, suffix heap size=140496,new segment heap size=29568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1082, new segment cells count=282 2023-03-27 04:58:35,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=282, cellSize=12912, totalHeapSize=29568, min timestamp=1679893115774, max timestamp=1679893115780Num uniques 1082; 2023-03-27 04:58:35,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49552, new segment data size=12912, suffix heap size=140496,new segment heap size=29568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1082, new segment cells count=282 2023-03-27 04:58:35,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=341, cellSize=15616, totalHeapSize=35704, min timestamp=1679893115789, max timestamp=1679893115794Num uniques 1141; 2023-03-27 04:58:35,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=52256, new segment data size=15616, suffix heap size=146632,new segment heap size=35704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1141, new segment cells count=341 2023-03-27 04:58:35,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=341, cellSize=15616, totalHeapSize=35704, min timestamp=1679893115789, max timestamp=1679893115794Num uniques 1141; 2023-03-27 04:58:35,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=52256, new segment data size=15616, suffix heap size=146632,new segment heap size=35704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1141, new segment cells count=341 2023-03-27 04:58:35,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=341, cellSize=15616, totalHeapSize=35704, min timestamp=1679893115789, max timestamp=1679893115794Num uniques 1141; 2023-03-27 04:58:35,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=52256, new segment data size=15616, suffix heap size=146632,new segment heap size=35704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1141, new segment cells count=341 2023-03-27 04:58:35,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=200, cellSize=9160, totalHeapSize=21040, min timestamp=1679893115825, max timestamp=1679893115833Num uniques 1200; 2023-03-27 04:58:35,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=54960, new segment data size=9160, suffix heap size=152768,new segment heap size=21040 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1200, new segment cells count=200 2023-03-27 04:58:35,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=200, cellSize=9160, totalHeapSize=21040, min timestamp=1679893115825, max timestamp=1679893115833Num uniques 1200; 2023-03-27 04:58:35,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=54960, new segment data size=9160, suffix heap size=152768,new segment heap size=21040 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1200, new segment cells count=200 2023-03-27 04:58:35,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=200, cellSize=9160, totalHeapSize=21040, min timestamp=1679893115825, max timestamp=1679893115833Num uniques 1200; 2023-03-27 04:58:35,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=54960, new segment data size=9160, suffix heap size=152768,new segment heap size=21040 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1200, new segment cells count=200 2023-03-27 04:58:35,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893115845, max timestamp=1679893115849Num uniques 1059; 2023-03-27 04:58:35,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48495, new segment data size=7275, suffix heap size=138104,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1059, new segment cells count=159 2023-03-27 04:58:35,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893115845, max timestamp=1679893115849Num uniques 1059; 2023-03-27 04:58:35,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893115845, max timestamp=1679893115849Num uniques 1059; 2023-03-27 04:58:35,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48495, new segment data size=7275, suffix heap size=138104,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1059, new segment cells count=159 2023-03-27 04:58:35,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48495, new segment data size=7275, suffix heap size=138104,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1059, new segment cells count=159 2023-03-27 04:58:35,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679893115862, max timestamp=1679893115866Num uniques 1018; 2023-03-27 04:58:35,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-27 04:58:35,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679893115862, max timestamp=1679893115866Num uniques 1018; 2023-03-27 04:58:35,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-27 04:58:35,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679893115862, max timestamp=1679893115866Num uniques 1018; 2023-03-27 04:58:35,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-27 04:58:35,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679893115877, max timestamp=1679893115900Num uniques 1077; 2023-03-27 04:58:35,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679893115877, max timestamp=1679893115900Num uniques 1077; 2023-03-27 04:58:35,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-27 04:58:35,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679893115877, max timestamp=1679893115900Num uniques 1077; 2023-03-27 04:58:35,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-27 04:58:35,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-27 04:58:35,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679893115913, max timestamp=1679893115920Num uniques 1086; 2023-03-27 04:58:35,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679893115913, max timestamp=1679893115920Num uniques 1086; 2023-03-27 04:58:35,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-27 04:58:35,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-27 04:58:35,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679893115913, max timestamp=1679893115920Num uniques 1086; 2023-03-27 04:58:35,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-27 04:58:35,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1679893115929, max timestamp=1679893115935Num uniques 1045; 2023-03-27 04:58:35,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1679893115929, max timestamp=1679893115935Num uniques 1045; 2023-03-27 04:58:35,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1679893115929, max timestamp=1679893115935Num uniques 1045; 2023-03-27 04:58:35,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=18090, suffix heap size=136648,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=395 2023-03-27 04:58:35,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=18090, suffix heap size=136648,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=395 2023-03-27 04:58:35,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=18090, suffix heap size=136648,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=395 2023-03-27 04:58:35,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679893115947, max timestamp=1679893115952Num uniques 1254; 2023-03-27 04:58:35,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=57430, new segment data size=11630, suffix heap size=158384,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=254 2023-03-27 04:58:35,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679893115947, max timestamp=1679893115952Num uniques 1254; 2023-03-27 04:58:35,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=57430, new segment data size=11630, suffix heap size=158384,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=254 2023-03-27 04:58:35,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679893115947, max timestamp=1679893115952Num uniques 1254; 2023-03-27 04:58:35,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=57430, new segment data size=11630, suffix heap size=158384,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=254 2023-03-27 04:58:35,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679893115985, max timestamp=1679893115991Num uniques 1113; 2023-03-27 04:58:35,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-27 04:58:35,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679893115985, max timestamp=1679893115991Num uniques 1113; 2023-03-27 04:58:35,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679893115985, max timestamp=1679893115991Num uniques 1113; 2023-03-27 04:58:35,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-27 04:58:35,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-27 04:58:36,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1679893116022, max timestamp=1679893116027Num uniques 1072; 2023-03-27 04:58:36,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-03-27 04:58:36,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1679893116022, max timestamp=1679893116027Num uniques 1072; 2023-03-27 04:58:36,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1679893116022, max timestamp=1679893116027Num uniques 1072; 2023-03-27 04:58:36,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-03-27 04:58:36,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-03-27 04:58:36,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679893116038, max timestamp=1679893116041Num uniques 1131; 2023-03-27 04:58:36,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=12866, suffix heap size=145592,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=281 2023-03-27 04:58:36,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679893116038, max timestamp=1679893116041Num uniques 1131; 2023-03-27 04:58:36,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=12866, suffix heap size=145592,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=281 2023-03-27 04:58:36,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679893116038, max timestamp=1679893116041Num uniques 1131; 2023-03-27 04:58:36,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=12866, suffix heap size=145592,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=281 2023-03-27 04:58:36,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1679893116048, max timestamp=1679893116053Num uniques 1140; 2023-03-27 04:58:36,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1679893116048, max timestamp=1679893116053Num uniques 1140; 2023-03-27 04:58:36,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=15570, suffix heap size=146528,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=340 2023-03-27 04:58:36,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=15570, suffix heap size=146528,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=340 2023-03-27 04:58:36,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1679893116048, max timestamp=1679893116053Num uniques 1140; 2023-03-27 04:58:36,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=15570, suffix heap size=146528,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=340 2023-03-27 04:58:36,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1679893116058, max timestamp=1679893116071Num uniques 1199; 2023-03-27 04:58:36,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1679893116058, max timestamp=1679893116071Num uniques 1199; 2023-03-27 04:58:36,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=13694, suffix heap size=152664,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=299 2023-03-27 04:58:36,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=13694, suffix heap size=152664,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=299 2023-03-27 04:58:36,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1679893116058, max timestamp=1679893116071Num uniques 1199; 2023-03-27 04:58:36,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=13694, suffix heap size=152664,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=299 2023-03-27 04:58:36,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11808, totalHeapSize=27072, min timestamp=1679893116092, max timestamp=1679893116097Num uniques 1158; 2023-03-27 04:58:36,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11808, totalHeapSize=27072, min timestamp=1679893116092, max timestamp=1679893116097Num uniques 1158; 2023-03-27 04:58:36,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=53028, new segment data size=11808, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-03-27 04:58:36,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679893116092, max timestamp=1679893116097Num uniques 1158; 2023-03-27 04:58:36,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=53028, new segment data size=6870, suffix heap size=148400,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=150 2023-03-27 04:58:36,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=53028, new segment data size=11808, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-03-27 04:58:36,104 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/f15cad0a2a424b2ea24de9bc3ac7932e 2023-03-27 04:58:36,109 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/e56de8e34969424bbdb26538d85fdb38 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/e56de8e34969424bbdb26538d85fdb38 2023-03-27 04:58:36,115 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/e56de8e34969424bbdb26538d85fdb38, entries=50, sequenceid=177, filesize=11.6 K 2023-03-27 04:58:36,116 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/1cbf8b2767b7414d9cfc13a612c08263 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/1cbf8b2767b7414d9cfc13a612c08263 2023-03-27 04:58:36,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1679893116108, max timestamp=1679893116117Num uniques 1117; 2023-03-27 04:58:36,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=12222, suffix heap size=144136,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=267 2023-03-27 04:58:36,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1679893116108, max timestamp=1679893116117Num uniques 1117; 2023-03-27 04:58:36,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=12222, suffix heap size=144136,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=267 2023-03-27 04:58:36,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1679893116108, max timestamp=1679893116117Num uniques 1009; 2023-03-27 04:58:36,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=12222, suffix heap size=132904,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=267 2023-03-27 04:58:36,122 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/1cbf8b2767b7414d9cfc13a612c08263, entries=50, sequenceid=177, filesize=7.1 K 2023-03-27 04:58:36,131 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/f15cad0a2a424b2ea24de9bc3ac7932e as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/f15cad0a2a424b2ea24de9bc3ac7932e 2023-03-27 04:58:36,134 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/f15cad0a2a424b2ea24de9bc3ac7932e, entries=50, sequenceid=177, filesize=7.1 K 2023-03-27 04:58:36,135 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=127.47 KB/130530 for a34144f135b39d6005ad383002fabe3a in 1273ms, sequenceid=177, compaction requested=true 2023-03-27 04:58:36,135 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:36,135 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:36,135 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:36,135 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:36,135 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,135 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:36,135 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:36,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:36,138 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=134.18 KB heapSize=374.86 KB 2023-03-27 04:58:36,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:36,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:36,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:36,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:36,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:36,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:36,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50916 deadline: 1679893176144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893176145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893176145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893176146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893176146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,155 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303270e0e3a1429ad4579a498d59b01f9f8cb to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303270e0e3a1429ad4579a498d59b01f9f8cb 2023-03-27 04:58:36,188 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,188 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:36,188 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,188 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:36,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:36,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24f3fb13[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:36,190 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:36,190 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,190 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,190 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:36,190 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:36,191 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:36,191 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:36,192 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:36,192 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,192 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:36,192 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:36,192 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:36,192 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:36,192 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:36,192 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:36,192 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:36,192 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:36,193 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:36,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:36,193 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,193 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:36,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:36,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:36,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:36,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,195 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:36,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,196 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:36,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,198 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:36,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:36,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:36,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:36,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:36,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:36,198 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:36,199 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:36,199 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:36,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:36,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:36,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50916 deadline: 1679893176249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893176249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893176250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893176250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893176250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,299 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:36,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:36,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:36,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:36,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:36,302 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:36,302 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:36,302 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:36,302 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:36,302 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,302 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,303 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:36,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:36,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:36,303 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,303 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,303 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:36,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:36,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:36,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:36,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:36,304 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:36,304 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:36,305 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:36,316 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:36,316 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:36,316 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:36,316 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:36,316 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:36,316 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:36,316 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:36,316 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:36,316 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,316 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:36,316 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:36,317 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,317 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:36,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:36,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:36,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:36,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:36,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:36,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:36,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e40d1bf[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:36,319 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:36,319 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,320 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,320 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:36,320 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:36,320 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,320 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:36,320 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:36,321 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,321 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:36,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:36,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:36,322 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:36,322 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:36,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:36,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:36,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:36,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:36,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:36,324 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:36,324 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:36,324 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:36,324 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:36,324 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:36,324 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,324 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,325 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:36,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:36,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:36,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:36,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:36,325 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:36,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:36,326 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:36,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,328 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:36,328 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:36,328 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:36,328 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:36,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:36,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:36,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:36,328 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,328 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,328 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,328 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,328 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,328 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,328 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:36,328 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:36,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:36,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:36,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:36,428 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:36,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:36,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:36,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:36,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:36,431 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:36,431 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:36,431 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:36,431 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:36,432 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,432 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,432 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,432 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:36,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:36,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:36,433 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,433 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:36,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:36,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:36,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:36,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:36,433 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:36,433 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:36,433 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:36,434 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:36,434 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:36,434 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:36,434 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:36,434 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:36,434 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:36,434 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:36,434 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:36,434 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,434 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:36,434 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:36,435 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:36,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:36,435 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:36,436 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,436 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:36,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:36,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ef2b991[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:36,437 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:36,437 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,438 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,438 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:36,438 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,439 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:36,438 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:36,439 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:36,439 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:36,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:36,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:36,440 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:36,440 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:36,440 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:36,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:36,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:36,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:36,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:36,443 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:36,443 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:36,443 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:36,443 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:36,443 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:36,444 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,444 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:36,444 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:36,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:36,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:36,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:36,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:36,444 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:36,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,445 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:36,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,447 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:36,447 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:36,447 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,447 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:36,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:36,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:36,447 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:36,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:36,448 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,448 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:36,448 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,448 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:36,448 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,448 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,448 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:36,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:36,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893176452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893176452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50916 deadline: 1679893176453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893176454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893176454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,548 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:36,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:36,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:36,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:36,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:36,550 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:36,550 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:36,551 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:36,551 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:36,551 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,551 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,551 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:36,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:36,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:36,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,551 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:36,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,553 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,553 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:36,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:36,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:36,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:36,553 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:36,553 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:36,553 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:36,554 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:36,554 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:36,554 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:36,554 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:36,554 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:36,554 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:36,554 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:36,554 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:36,554 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,554 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:36,554 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:36,555 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,555 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:36,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:36,555 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,555 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:36,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:36,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:36,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:36,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:36,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a097057[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:36,556 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:36,556 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,557 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,557 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:36,557 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:36,557 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,557 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:36,557 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:36,558 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:36,558 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:36,558 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:36,558 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:36,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:36,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:36,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:36,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:36,560 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:36,560 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:36,560 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:36,560 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:36,560 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:36,561 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,561 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,561 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:36,561 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:36,562 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:36,563 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,563 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,563 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,563 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:36,563 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:36,563 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:36,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:36,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:36,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:36,563 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,563 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:36,564 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:36,564 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:36,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:36,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:36,564 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:36,564 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,564 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1278, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/750129795cd24615922209900dcf1c74 2023-03-27 04:58:36,576 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1278 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/020591f377da475289a95065cba92fe9 2023-03-27 04:58:36,619 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1278 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/ad6064b584e04e24b198ed0b87bdce5f 2023-03-27 04:58:36,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/750129795cd24615922209900dcf1c74 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/750129795cd24615922209900dcf1c74 2023-03-27 04:58:36,625 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/750129795cd24615922209900dcf1c74, entries=200, sequenceid=1278, filesize=32.3 K 2023-03-27 04:58:36,625 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/020591f377da475289a95065cba92fe9 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/020591f377da475289a95065cba92fe9 2023-03-27 04:58:36,633 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/020591f377da475289a95065cba92fe9, entries=150, sequenceid=1278, filesize=11.9 K 2023-03-27 04:58:36,633 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/ad6064b584e04e24b198ed0b87bdce5f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ad6064b584e04e24b198ed0b87bdce5f 2023-03-27 04:58:36,636 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ad6064b584e04e24b198ed0b87bdce5f, entries=150, sequenceid=1278, filesize=11.9 K 2023-03-27 04:58:36,636 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~374.81 KB/383808, currentSize=53.67 KB/54960 for a34144f135b39d6005ad383002fabe3a in 498ms, sequenceid=1278, compaction requested=true 2023-03-27 04:58:36,636 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:36,636 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:36,636 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:58:36,636 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:58:36,636 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:36,636 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:36,637 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 97092 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:58:36,637 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/A is initiating minor compaction (all files) 2023-03-27 04:58:36,637 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/A in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:36,638 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/c798d48c7dcf4d55a90620c4cf27ee1e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/51fe687a6eae4ddbbaf2168dd462ebb5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/e56de8e34969424bbdb26538d85fdb38, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/750129795cd24615922209900dcf1c74] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=94.8 K 2023-03-27 04:58:36,638 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 43850 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:58:36,638 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/B is initiating minor compaction (all files) 2023-03-27 04:58:36,638 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/B in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:36,638 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/369ffe9420df42cba7a87022c882f6bf, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/0df06ca66f8544a0927983c60bb35fb3, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/1cbf8b2767b7414d9cfc13a612c08263, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/020591f377da475289a95065cba92fe9] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=42.8 K 2023-03-27 04:58:36,638 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting c798d48c7dcf4d55a90620c4cf27ee1e, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1679893111578 2023-03-27 04:58:36,638 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 51fe687a6eae4ddbbaf2168dd462ebb5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1679893112702 2023-03-27 04:58:36,638 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 369ffe9420df42cba7a87022c882f6bf, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1679893111578 2023-03-27 04:58:36,639 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting e56de8e34969424bbdb26538d85fdb38, keycount=50, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1679893114861 2023-03-27 04:58:36,639 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 0df06ca66f8544a0927983c60bb35fb3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1679893112702 2023-03-27 04:58:36,639 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 750129795cd24615922209900dcf1c74, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1278, earliestPutTs=1679893116130 2023-03-27 04:58:36,640 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 1cbf8b2767b7414d9cfc13a612c08263, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1679893114861 2023-03-27 04:58:36,640 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 020591f377da475289a95065cba92fe9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1278, earliestPutTs=1679893116130 2023-03-27 04:58:36,646 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#B#compaction#504 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:36,649 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#A#compaction#503 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:36,664 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:36,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:36,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:36,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:36,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:36,670 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:36,670 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:36,670 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:36,670 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:36,670 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,670 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,671 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:36,671 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:36,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:36,671 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,671 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:36,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:36,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:36,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:36,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:36,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:36,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:36,673 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:36,673 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:36,673 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:36,683 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:36,683 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:36,683 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:36,683 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:36,683 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-27 04:58:36,684 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:36,684 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:36,684 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:36,684 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:36,684 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:36,684 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:36,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:36,759 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:36,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893176772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893116770, max timestamp=1679893116776Num uniques 859; 2023-03-27 04:58:36,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893116770, max timestamp=1679893116776Num uniques 859; 2023-03-27 04:58:36,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:58:36,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893176772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:58:36,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893116770, max timestamp=1679893116776Num uniques 859; 2023-03-27 04:58:36,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:58:36,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50916 deadline: 1679893176801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893176804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893176804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893176879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893176879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50916 deadline: 1679893176906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893176908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:36,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:36,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893176908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:37,068 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/b8810a58181b4936bf7a8cf5a71e1d4b as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/b8810a58181b4936bf7a8cf5a71e1d4b 2023-03-27 04:58:37,072 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a34144f135b39d6005ad383002fabe3a/B of a34144f135b39d6005ad383002fabe3a into b8810a58181b4936bf7a8cf5a71e1d4b(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:37,072 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:37,072 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/B, priority=12, startTime=1679893116636; duration=0sec 2023-03-27 04:58:37,072 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:37,072 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:58:37,073 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 43850 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:58:37,073 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/C is initiating minor compaction (all files) 2023-03-27 04:58:37,073 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/C in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:37,073 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/20d6c2e543294ba3bd9c0ebe38fcd4a6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/83998e74701a49c99519f06c62636cf5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/f15cad0a2a424b2ea24de9bc3ac7932e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ad6064b584e04e24b198ed0b87bdce5f] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=42.8 K 2023-03-27 04:58:37,074 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 20d6c2e543294ba3bd9c0ebe38fcd4a6, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1679893111578 2023-03-27 04:58:37,074 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 83998e74701a49c99519f06c62636cf5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1679893112702 2023-03-27 04:58:37,074 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting f15cad0a2a424b2ea24de9bc3ac7932e, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1679893114861 2023-03-27 04:58:37,074 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting ad6064b584e04e24b198ed0b87bdce5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1278, earliestPutTs=1679893116130 2023-03-27 04:58:37,081 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#C#compaction#506 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:37,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:37,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893177083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:37,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:37,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893177083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:37,090 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/dadea9052e354c4aaafdadd459fe2602 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/dadea9052e354c4aaafdadd459fe2602 2023-03-27 04:58:37,094 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a34144f135b39d6005ad383002fabe3a/C of a34144f135b39d6005ad383002fabe3a into dadea9052e354c4aaafdadd459fe2602(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:37,094 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:37,094 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/C, priority=12, startTime=1679893116636; duration=0sec 2023-03-27 04:58:37,094 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:37,116 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303279160337b7d814feeb383ab23e02311f0 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303279160337b7d814feeb383ab23e02311f0 2023-03-27 04:58:37,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:37,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50916 deadline: 1679893177109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:37,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:37,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893177110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:37,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:37,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893177111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:37,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:37,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893177384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:37,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:37,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893177388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:37,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:37,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50916 deadline: 1679893177417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:37,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:37,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893177417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:37,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:37,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893177419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:37,491 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/5af325f076ba4cf7a776db0609c314c6 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/5af325f076ba4cf7a776db0609c314c6 2023-03-27 04:58:37,495 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a34144f135b39d6005ad383002fabe3a/A of a34144f135b39d6005ad383002fabe3a into 5af325f076ba4cf7a776db0609c314c6(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:37,495 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:37,495 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/A, priority=12, startTime=1679893116636; duration=0sec 2023-03-27 04:58:37,495 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:37,532 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1289, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/441dd5c3b844405394d811634358c4b0 2023-03-27 04:58:37,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:37,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893177885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:37,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:37,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893177890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:37,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:37,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50916 deadline: 1679893177920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:37,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:37,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893177922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:37,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:37,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893177922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:37,952 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1289 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/792a8834566e4026999d13fd874d99b2 2023-03-27 04:58:38,361 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1289 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/a49914e25d7247af8f173e2509499c0d 2023-03-27 04:58:38,364 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/441dd5c3b844405394d811634358c4b0 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/441dd5c3b844405394d811634358c4b0 2023-03-27 04:58:38,367 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/441dd5c3b844405394d811634358c4b0, entries=150, sequenceid=1289, filesize=25.4 K 2023-03-27 04:58:38,368 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/792a8834566e4026999d13fd874d99b2 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/792a8834566e4026999d13fd874d99b2 2023-03-27 04:58:38,371 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/792a8834566e4026999d13fd874d99b2, entries=150, sequenceid=1289, filesize=11.9 K 2023-03-27 04:58:38,371 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/a49914e25d7247af8f173e2509499c0d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/a49914e25d7247af8f173e2509499c0d 2023-03-27 04:58:38,373 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/a49914e25d7247af8f173e2509499c0d, entries=150, sequenceid=1289, filesize=11.9 K 2023-03-27 04:58:38,374 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for a34144f135b39d6005ad383002fabe3a in 1691ms, sequenceid=1289, compaction requested=false 2023-03-27 04:58:38,374 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:38,374 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:38,374 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:38,374 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:38,374 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,374 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:38,374 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:38,375 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:38,375 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:38,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:38,375 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:38,375 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:38,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:38,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:38,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:38,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:38,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:38,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21992b66[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:38,377 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:38,377 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:38,377 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:38,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:38,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:38,377 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:38,377 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:38,377 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:38,377 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,377 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:38,378 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:38,378 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:38,378 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,378 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:38,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:38,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:38,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:38,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:38,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:38,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:38,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:38,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:38,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:38,380 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:38,380 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:38,380 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:38,380 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:38,380 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:38,381 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:38,381 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:38,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:38,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:38,381 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:38,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:38,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:38,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:38,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:38,381 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:38,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:38,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:38,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:38,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:38,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:38,382 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,384 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:38,384 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:38,384 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:38,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:38,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:38,384 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:38,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:38,384 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:38,384 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:38,385 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:38,384 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:38,385 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,385 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:38,385 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:38,385 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:38,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:38,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:38,485 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:38,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:38,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:38,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:38,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:38,487 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:38,487 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:38,487 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:38,487 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:38,488 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:38,488 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:38,488 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:38,488 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:38,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:38,488 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,489 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:38,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:38,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:38,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:38,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:38,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:38,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:38,489 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:38,489 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:38,489 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:38,489 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:38,489 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:38,489 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:38,489 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:38,490 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:38,490 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-27 04:58:38,490 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:38,490 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:38,490 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:38,490 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:38,490 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:38,490 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:38,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:38,896 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:38,903 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327fc013c80436e4f8b84a16a495a74d370 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327fc013c80436e4f8b84a16a495a74d370 2023-03-27 04:58:38,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:38,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893178925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:38,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893178929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:38,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893178929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:38,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50916 deadline: 1679893178930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:38,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:38,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893178934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:39,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:39,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893179033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:39,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:39,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893179033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:39,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:39,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893179237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:39,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:39,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893179238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:39,307 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1328, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/f2e6edd98aba46a1bae0bc55e2b332a4 2023-03-27 04:58:39,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:39,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893179541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:39,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:39,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893179541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:39,715 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1328 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/0fa5c70f80414a62b86e84231dbf9531 2023-03-27 04:58:40,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:40,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893180045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:40,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893180048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,144 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1328 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/a922eb1d75554ac7a9aeaa6b83e886f4 2023-03-27 04:58:40,148 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/f2e6edd98aba46a1bae0bc55e2b332a4 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/f2e6edd98aba46a1bae0bc55e2b332a4 2023-03-27 04:58:40,152 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/f2e6edd98aba46a1bae0bc55e2b332a4, entries=150, sequenceid=1328, filesize=25.4 K 2023-03-27 04:58:40,153 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/0fa5c70f80414a62b86e84231dbf9531 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/0fa5c70f80414a62b86e84231dbf9531 2023-03-27 04:58:40,157 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/0fa5c70f80414a62b86e84231dbf9531, entries=150, sequenceid=1328, filesize=11.9 K 2023-03-27 04:58:40,157 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/a922eb1d75554ac7a9aeaa6b83e886f4 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/a922eb1d75554ac7a9aeaa6b83e886f4 2023-03-27 04:58:40,161 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/a922eb1d75554ac7a9aeaa6b83e886f4, entries=150, sequenceid=1328, filesize=11.9 K 2023-03-27 04:58:40,162 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for a34144f135b39d6005ad383002fabe3a in 1672ms, sequenceid=1328, compaction requested=true 2023-03-27 04:58:40,162 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:40,162 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:40,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:40,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:40,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:40,162 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:40,166 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,166 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:40,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:40,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,167 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:40,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:40,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@394ccbc3[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:40,168 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:40,168 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,173 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,173 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:40,173 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:40,173 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:40,173 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,173 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:40,174 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:40,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:40,174 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:40,175 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:40,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:40,175 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:40,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:40,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:40,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:40,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:40,177 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:40,177 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:40,177 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:40,177 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:40,177 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:40,179 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,180 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:40,180 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:40,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:40,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:40,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:40,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:40,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:40,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,182 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:40,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,189 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:40,190 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:40,190 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,190 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,190 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:40,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:40,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:40,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:40,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:40,190 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,190 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,190 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,190 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,190 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,190 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:40,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:40,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:40,190 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:40,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:40,290 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:40,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:40,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:40,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:40,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:40,311 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:40,311 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:40,311 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:40,311 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:40,311 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,311 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,312 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:40,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:40,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:40,312 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,313 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,313 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:40,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:40,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:40,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:40,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:40,326 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:40,326 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:40,326 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:40,330 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:40,330 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:40,330 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:40,330 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:40,330 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:58:40,330 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:40,330 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:40,330 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:40,330 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:40,331 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:40,331 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:40,393 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032753849835d30547d3a323fef1eab93ee3 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032753849835d30547d3a323fef1eab93ee3 2023-03-27 04:58:40,428 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1338, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/bbca2454412a4cc19b82dce057cdf9e3 2023-03-27 04:58:40,473 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1338 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/f38008770ff6487d949e4c58cb21625f 2023-03-27 04:58:40,520 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1338 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/5944e6028c8147e38623498945b749a7 2023-03-27 04:58:40,526 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/bbca2454412a4cc19b82dce057cdf9e3 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/bbca2454412a4cc19b82dce057cdf9e3 2023-03-27 04:58:40,531 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/bbca2454412a4cc19b82dce057cdf9e3, entries=150, sequenceid=1338, filesize=25.4 K 2023-03-27 04:58:40,533 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/f38008770ff6487d949e4c58cb21625f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/f38008770ff6487d949e4c58cb21625f 2023-03-27 04:58:40,540 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/f38008770ff6487d949e4c58cb21625f, entries=150, sequenceid=1338, filesize=11.9 K 2023-03-27 04:58:40,541 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/5944e6028c8147e38623498945b749a7 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/5944e6028c8147e38623498945b749a7 2023-03-27 04:58:40,546 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/5944e6028c8147e38623498945b749a7, entries=150, sequenceid=1338, filesize=11.9 K 2023-03-27 04:58:40,547 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for a34144f135b39d6005ad383002fabe3a in 217ms, sequenceid=1338, compaction requested=true 2023-03-27 04:58:40,547 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:40,547 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:40,547 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:40,547 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:40,547 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,547 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:40,547 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:40,548 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,548 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:40,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:40,549 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,549 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:40,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:40,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:40,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:40,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:40,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7239c34a[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:40,550 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:40,550 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,551 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,551 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:40,558 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,558 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:40,558 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:40,558 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:40,564 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:40,564 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,564 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:40,565 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:40,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:40,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:40,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:40,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:40,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:40,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:40,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:40,569 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:40,569 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:40,569 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:40,569 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:40,569 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:40,573 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,573 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:40,573 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:40,573 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:40,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,578 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:40,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,585 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:40,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:40,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:40,586 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:40,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:40,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:40,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:40,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,586 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:40,587 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:40,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:40,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:40,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:40,687 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:40,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:40,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:40,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:40,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:40,691 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:40,691 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:40,691 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:40,691 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:40,692 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,692 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,692 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,693 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,693 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:40,694 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:40,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:40,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:40,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:40,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:40,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:40,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:40,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:40,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:40,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:40,696 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:40,696 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:40,696 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:40,696 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:40,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:40,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:40,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,697 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:40,697 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:40,697 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,698 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:40,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:40,698 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,698 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:40,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:40,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:40,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:40,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:40,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23638e92[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:40,700 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:40,700 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,701 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,701 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:40,701 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:40,701 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:40,701 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,701 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:40,702 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,702 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:40,702 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:40,702 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:40,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:40,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:40,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:40,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:40,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:40,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:40,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:40,704 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:40,704 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:40,705 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:40,705 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:40,705 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:40,706 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,706 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:40,706 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:40,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:40,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:40,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:40,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:40,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,707 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:40,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,708 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:40,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,711 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:40,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:40,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:40,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:40,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:40,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:40,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:40,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:40,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:40,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:40,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,711 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:40,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,711 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:40,812 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:40,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:40,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:40,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:40,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:40,815 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:40,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:40,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:40,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:40,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,815 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,816 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:40,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:40,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:40,816 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,816 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,816 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:40,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:40,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:40,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:40,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:40,817 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:40,817 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:40,817 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:40,817 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:40,817 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:40,817 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:40,817 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:40,817 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:40,817 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:40,817 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:40,817 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:40,817 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,818 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:40,818 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:40,818 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,818 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:40,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:40,819 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,819 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:40,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:40,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:40,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:40,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:40,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b494383[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:40,820 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:40,820 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,821 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,821 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:40,821 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:40,821 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,821 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:40,821 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:40,821 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,821 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:40,822 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:40,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:40,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:40,822 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:40,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:40,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:40,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:40,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:40,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:40,823 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:40,823 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:40,823 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:40,824 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:40,824 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:40,824 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,824 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:40,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:40,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,824 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:40,825 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:40,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:40,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:40,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:40,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,826 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:40,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:40,829 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:40,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:40,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:40,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:40,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:40,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:40,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:40,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:40,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:40,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,829 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:40,829 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:40,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,929 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:40,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:40,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:40,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:40,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:40,932 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:40,932 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:40,932 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:40,932 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:40,933 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,933 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,934 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:40,934 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:40,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:40,934 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,934 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:40,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:40,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:40,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:40,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:40,935 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:40,935 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:40,935 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:40,936 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:40,936 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:40,936 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:40,937 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:40,937 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:40,937 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:40,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:40,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:40,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:40,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:40,939 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,939 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:40,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:40,940 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,940 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:40,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:40,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:40,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:40,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:40,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14048bfa[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:40,941 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:40,941 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,942 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,942 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:40,942 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:40,942 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,942 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:40,942 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:40,943 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,943 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:40,943 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:40,943 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:40,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:40,945 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:40,945 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:40,945 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:40,945 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:40,945 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:40,946 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,946 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:40,946 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:40,946 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,947 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:40,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:40,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,949 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,949 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:40,949 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,949 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:40,949 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:40,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:40,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:40,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:40,949 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:40,949 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,949 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:40,949 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:40,949 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:40,949 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:40,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:40,949 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:40,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:40,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:40,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:40,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:40,966 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:58:40,966 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:40,966 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:40,966 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:40,967 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:40,967 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:40,967 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:40,985 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327d19017a1306c4e17969b757b1df8de9d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327d19017a1306c4e17969b757b1df8de9d 2023-03-27 04:58:41,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893121005, max timestamp=1679893121012Num uniques 859; 2023-03-27 04:58:41,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893121005, max timestamp=1679893121012Num uniques 859; 2023-03-27 04:58:41,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679893121005, max timestamp=1679893121012Num uniques 859; 2023-03-27 04:58:41,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:58:41,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:58:41,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-27 04:58:41,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:41,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893181033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:41,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50916 deadline: 1679893181035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:41,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893181036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,049 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:41,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:41,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:41,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:41,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:41,052 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:41,052 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:41,052 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:41,052 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:41,052 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,052 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,053 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:41,053 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:41,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:41,053 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,053 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:41,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,054 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:41,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:41,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:41,054 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:41,054 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:41,054 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:41,054 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:41,054 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:41,054 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:41,054 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:41,054 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:41,054 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:41,054 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:41,054 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:41,054 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,055 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:41,055 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:41,055 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,055 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,055 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,055 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:41,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:41,057 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:41,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@289b39a3[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:41,057 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:41,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893181056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,057 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:41,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893181056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,058 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:41,058 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:41,058 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:41,058 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,058 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:41,058 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:41,058 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:41,058 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,058 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:41,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:41,060 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:41,060 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:41,060 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:41,060 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:41,060 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:41,061 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,061 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,061 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:41,062 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:41,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:41,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,062 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,064 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,064 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:41,064 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:41,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:41,064 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:41,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:41,064 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:41,064 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:41,064 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:41,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:41,064 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,064 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:41,064 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,065 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,065 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:41,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:41,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:41,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893181136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:41,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50916 deadline: 1679893181138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:41,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893181138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,164 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:41,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:41,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:41,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:41,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:41,167 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:41,167 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:41,167 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:41,167 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:41,168 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,168 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,168 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:41,168 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:41,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:41,169 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,169 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:41,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:41,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:41,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:41,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:41,169 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:41,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:41,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:41,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:41,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:41,170 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:41,170 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:41,170 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:41,170 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:41,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:41,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:41,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:41,170 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:41,171 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,171 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,171 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,171 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:41,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1733253c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:41,172 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:41,172 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,173 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,173 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:41,173 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:41,173 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:41,173 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,173 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:41,173 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,173 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,173 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:41,174 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:41,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:41,175 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:41,175 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:41,175 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:41,175 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:41,175 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:41,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,176 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:41,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:41,176 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,177 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,180 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,180 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,180 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:41,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:41,180 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:41,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:41,180 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,180 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:41,180 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:41,180 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:41,180 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,180 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,180 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:41,180 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:41,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:41,292 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:41,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:41,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:41,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:41,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:41,304 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:41,304 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:41,304 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:41,304 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:41,304 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,304 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,305 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,305 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:41,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:41,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:41,305 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,305 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:41,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:41,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:41,306 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:41,306 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:41,306 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:41,310 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:41,310 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:41,310 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:41,310 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:41,310 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:41,310 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:41,310 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:41,310 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:41,310 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,310 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:41,310 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:41,313 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,313 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,313 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,313 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:41,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16672902[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:41,314 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:41,315 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,317 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,317 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:41,317 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:41,317 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:41,317 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,317 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:41,318 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:41,319 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,319 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:41,319 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:41,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:41,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:41,321 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:41,321 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:41,321 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:41,321 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:41,321 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:41,321 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,321 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,322 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:41,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,323 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,324 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,325 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,325 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,325 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,325 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:41,325 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:41,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:41,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:41,325 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,325 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,325 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,325 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,325 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:41,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:41,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:41,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:41,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:41,326 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:41,326 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:41,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:41,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:41,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893181338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:41,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50916 deadline: 1679893181342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:41,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893181342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,390 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1351, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/8c4347c1f03e47b4a4710db33e0cda30 2023-03-27 04:58:41,426 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:41,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:41,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:41,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:41,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:41,430 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:41,430 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:41,430 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:41,430 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:41,431 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,431 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,431 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:41,431 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:41,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:41,431 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,432 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:41,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:41,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:41,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:41,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:41,433 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:41,433 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:41,433 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:41,433 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:41,433 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:41,433 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:41,433 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:41,433 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:41,433 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:41,433 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:41,433 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:41,434 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,434 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:41,434 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:41,437 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,437 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,437 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,437 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:41,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:41,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@260c2b6f[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:41,440 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:41,440 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,440 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,440 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:41,440 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:41,440 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:41,441 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,441 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:41,441 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:41,441 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:41,441 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,441 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:41,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:41,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:41,446 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:41,446 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:41,446 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:41,446 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:41,446 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:41,452 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,452 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:41,452 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:41,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:41,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,454 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,455 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:41,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:41,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:41,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:41,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:41,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:41,461 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:41,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:41,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:41,461 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:41,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:41,461 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:41,562 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:41,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:41,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:41,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:41,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:41,566 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:41,566 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:41,566 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:41,566 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:41,566 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,566 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,567 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,567 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:41,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:41,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:41,567 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,567 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:41,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:41,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:41,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:41,569 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:41,569 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:41,569 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:41,569 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:41,569 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:41,569 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:41,569 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:41,569 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:41,569 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:41,569 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:41,570 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:41,570 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,570 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:41,570 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:41,570 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,570 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,571 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,571 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:41,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@572b582c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:41,572 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:41,572 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,573 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,573 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:41,573 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:41,573 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:41,573 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,573 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:41,574 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,574 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,574 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:41,574 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:41,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:41,576 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:41,576 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:41,576 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:41,576 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:41,576 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:41,577 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,577 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,577 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:41,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,577 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:41,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:41,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,578 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,589 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:41,589 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:41,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:41,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:41,590 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,590 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:41,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:41,590 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:41,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:41,590 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:41,590 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,590 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,590 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,590 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,590 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:41,590 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:41,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:41,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893181644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:41,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50916 deadline: 1679893181646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:41,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893181648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,690 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:41,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:41,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:41,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:41,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:41,694 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:41,694 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:41,694 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:41,694 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:41,695 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,695 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,697 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,697 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:41,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:41,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:41,697 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,697 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:41,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:41,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:41,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:41,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:41,699 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:41,699 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:41,699 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:41,715 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:41,715 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:41,715 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:41,716 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:41,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:41,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:41,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:41,716 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:41,717 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,717 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,717 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,717 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:41,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59bf8d30[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:41,718 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:41,718 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,719 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:41,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:41,719 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,719 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:41,719 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:41,720 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,720 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:41,720 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:41,720 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:41,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:41,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:41,722 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:41,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:41,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:41,722 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:41,722 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:41,723 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,723 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,723 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:41,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:41,724 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,725 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,727 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,727 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,727 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,727 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,727 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,727 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,727 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,727 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:41,727 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:41,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:41,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:41,727 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:41,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:41,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:41,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:41,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:41,727 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:41,727 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:41,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:41,804 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1351 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/f1843cdd8e62467db3a3bc671ff8882a 2023-03-27 04:58:41,828 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:41,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:41,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:41,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:41,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:41,834 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:41,834 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:41,834 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:41,834 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:41,834 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,835 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,835 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,836 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,836 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:41,836 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:41,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:41,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:41,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:41,850 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:41,850 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:41,850 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:41,850 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:41,850 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:41,850 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:41,850 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:41,851 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:41,851 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:41,851 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:41,851 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:41,851 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,851 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:41,851 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:41,852 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,852 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,852 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,852 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:41,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:41,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76bf803c[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:41,853 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:41,853 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,854 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,854 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:41,854 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:41,854 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:41,855 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,855 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:41,855 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:41,855 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:41,855 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,855 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:41,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:41,858 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:41,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:41,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:41,858 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:41,858 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:41,859 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,859 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,859 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:41,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:41,861 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,862 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,869 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:41,869 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,869 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,869 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,869 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,869 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,869 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,869 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,869 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:41,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:41,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:41,869 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:41,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:41,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:41,870 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:41,870 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:41,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:41,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:41,970 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:41,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:41,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:41,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:41,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:41,986 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:41,986 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:41,986 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:41,986 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:41,986 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,986 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,987 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:41,987 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:41,987 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,987 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:41,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:41,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:41,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:41,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:41,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:41,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:41,989 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:41,989 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:41,989 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:41,994 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:41,994 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:41,994 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:41,994 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:41,995 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,995 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,996 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:41,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:41,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:41,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31af9ad7[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:41,997 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:41,997 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,998 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:41,998 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:41,998 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:41,998 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:41,998 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,998 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:41,999 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:41,999 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:41,999 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:41,999 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:42,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:42,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:42,002 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:42,002 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:42,002 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:42,002 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:42,002 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:42,003 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,003 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,003 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,003 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:42,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:42,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:42,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:42,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:42,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:42,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:42,004 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:42,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:42,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:42,006 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:42,006 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:42,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:42,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:42,006 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,007 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:42,007 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:42,007 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,007 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,007 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,007 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:42,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:42,006 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:42,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:42,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:42,007 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:42,007 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:42,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:42,107 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:42,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:42,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:42,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:42,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:42,118 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:42,118 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:42,118 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:42,118 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:42,119 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,119 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:42,119 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:42,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:42,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:42,120 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:42,120 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,120 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:42,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:42,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:42,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:42,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:42,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:42,122 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:42,122 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:42,122 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:42,130 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:42,130 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:42,130 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:42,130 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:42,130 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:42,130 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:42,131 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:42,131 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:42,131 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,131 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:42,131 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:42,136 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,136 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:42,136 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,136 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:42,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:42,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c485ef6[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:42,138 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:42,138 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,139 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,140 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,140 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:42,140 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:42,140 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:42,140 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:42,141 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,141 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:42,141 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:42,141 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:42,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:42,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:42,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:42,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:42,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:42,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:42,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:42,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:42,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:42,152 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:42,152 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:42,152 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:42,152 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:42,152 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:42,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:42,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893182151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,154 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,154 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,155 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,154 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:42,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:42,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:42,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893182152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:42,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:42,156 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:42,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:42,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:42,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,158 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,158 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:42,158 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:42,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:42,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:42,158 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:42,158 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:42,158 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:42,158 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:42,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:42,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:42,158 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:42,159 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:42,158 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,159 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,159 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:42,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:42,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:42,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50916 deadline: 1679893182155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,232 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1351 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/9f2ecc4b3d434fe0947ebe901f697cae 2023-03-27 04:58:42,237 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/8c4347c1f03e47b4a4710db33e0cda30 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/8c4347c1f03e47b4a4710db33e0cda30 2023-03-27 04:58:42,242 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/8c4347c1f03e47b4a4710db33e0cda30, entries=350, sequenceid=1351, filesize=52.9 K 2023-03-27 04:58:42,243 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/f1843cdd8e62467db3a3bc671ff8882a as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/f1843cdd8e62467db3a3bc671ff8882a 2023-03-27 04:58:42,248 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/f1843cdd8e62467db3a3bc671ff8882a, entries=150, sequenceid=1351, filesize=11.9 K 2023-03-27 04:58:42,249 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/9f2ecc4b3d434fe0947ebe901f697cae as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/9f2ecc4b3d434fe0947ebe901f697cae 2023-03-27 04:58:42,254 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/9f2ecc4b3d434fe0947ebe901f697cae, entries=150, sequenceid=1351, filesize=11.9 K 2023-03-27 04:58:42,255 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a34144f135b39d6005ad383002fabe3a in 1289ms, sequenceid=1351, compaction requested=true 2023-03-27 04:58:42,255 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:42,255 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:42,255 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-27 04:58:42,255 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-27 04:58:42,255 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:42,255 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:42,256 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 158842 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-27 04:58:42,257 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/A is initiating minor compaction (all files) 2023-03-27 04:58:42,257 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/A in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:42,257 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/5af325f076ba4cf7a776db0609c314c6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/441dd5c3b844405394d811634358c4b0, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/f2e6edd98aba46a1bae0bc55e2b332a4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/bbca2454412a4cc19b82dce057cdf9e3, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/8c4347c1f03e47b4a4710db33e0cda30] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=155.1 K 2023-03-27 04:58:42,257 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 5af325f076ba4cf7a776db0609c314c6, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=1278, earliestPutTs=1679893116130 2023-03-27 04:58:42,258 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61408 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-27 04:58:42,258 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/B is initiating minor compaction (all files) 2023-03-27 04:58:42,258 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/B in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:42,258 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/b8810a58181b4936bf7a8cf5a71e1d4b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/792a8834566e4026999d13fd874d99b2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/0fa5c70f80414a62b86e84231dbf9531, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/f38008770ff6487d949e4c58cb21625f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/f1843cdd8e62467db3a3bc671ff8882a] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=60.0 K 2023-03-27 04:58:42,258 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 441dd5c3b844405394d811634358c4b0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1289, earliestPutTs=1679893116141 2023-03-27 04:58:42,259 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:42,259 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting b8810a58181b4936bf7a8cf5a71e1d4b, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=1278, earliestPutTs=1679893116130 2023-03-27 04:58:42,259 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting f2e6edd98aba46a1bae0bc55e2b332a4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1328, earliestPutTs=1679893116800 2023-03-27 04:58:42,259 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 792a8834566e4026999d13fd874d99b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1289, earliestPutTs=1679893116141 2023-03-27 04:58:42,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:42,260 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting bbca2454412a4cc19b82dce057cdf9e3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1338, earliestPutTs=1679893118903 2023-03-27 04:58:42,260 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 0fa5c70f80414a62b86e84231dbf9531, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1328, earliestPutTs=1679893116800 2023-03-27 04:58:42,261 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting f38008770ff6487d949e4c58cb21625f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1338, earliestPutTs=1679893118903 2023-03-27 04:58:42,261 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 8c4347c1f03e47b4a4710db33e0cda30, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=1351, earliestPutTs=1679893120953 2023-03-27 04:58:42,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:42,263 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting f1843cdd8e62467db3a3bc671ff8882a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1351, earliestPutTs=1679893120961 2023-03-27 04:58:42,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:42,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:42,270 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:42,270 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:42,270 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:42,270 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:42,271 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,271 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:42,271 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:42,271 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:42,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:42,272 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,272 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:42,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:42,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:42,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:42,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:42,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:42,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:42,281 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:42,281 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:42,281 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:42,283 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#B#compaction#518 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:42,289 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:42,289 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:42,289 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:42,289 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:42,289 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-27 04:58:42,290 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:42,290 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:42,290 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:42,290 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:42,290 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:42,290 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:42,356 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#A#compaction#519 average throughput is 0.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:42,414 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303279ce9a1c6ff9e412195a238487a46555b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303279ce9a1c6ff9e412195a238487a46555b 2023-03-27 04:58:42,457 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1387, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/d70fbc41bdb44fdda644cc6826a2fa12 2023-03-27 04:58:42,485 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1387 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/6bf05e3e05554519bde8f5e792fce413 2023-03-27 04:58:42,740 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/518e6513e7d443e1a7bcc154e3a50347 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/518e6513e7d443e1a7bcc154e3a50347 2023-03-27 04:58:42,744 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a34144f135b39d6005ad383002fabe3a/B of a34144f135b39d6005ad383002fabe3a into 518e6513e7d443e1a7bcc154e3a50347(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:42,744 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:42,744 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/B, priority=11, startTime=1679893122255; duration=0sec 2023-03-27 04:58:42,745 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:42,745 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-27 04:58:42,746 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61408 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-27 04:58:42,746 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/C is initiating minor compaction (all files) 2023-03-27 04:58:42,746 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/C in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:42,747 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/dadea9052e354c4aaafdadd459fe2602, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/a49914e25d7247af8f173e2509499c0d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/a922eb1d75554ac7a9aeaa6b83e886f4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/5944e6028c8147e38623498945b749a7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/9f2ecc4b3d434fe0947ebe901f697cae] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=60.0 K 2023-03-27 04:58:42,747 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting dadea9052e354c4aaafdadd459fe2602, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=1278, earliestPutTs=1679893116130 2023-03-27 04:58:42,747 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting a49914e25d7247af8f173e2509499c0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1289, earliestPutTs=1679893116141 2023-03-27 04:58:42,747 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting a922eb1d75554ac7a9aeaa6b83e886f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1328, earliestPutTs=1679893116800 2023-03-27 04:58:42,748 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 5944e6028c8147e38623498945b749a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1338, earliestPutTs=1679893118903 2023-03-27 04:58:42,748 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 9f2ecc4b3d434fe0947ebe901f697cae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1351, earliestPutTs=1679893120961 2023-03-27 04:58:42,756 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#C#compaction#523 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:42,786 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/5764f3f6e6334627ba7b6b74dc063910 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/5764f3f6e6334627ba7b6b74dc063910 2023-03-27 04:58:42,792 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a34144f135b39d6005ad383002fabe3a/C of a34144f135b39d6005ad383002fabe3a into 5764f3f6e6334627ba7b6b74dc063910(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:42,792 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:42,792 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/C, priority=11, startTime=1679893122255; duration=0sec 2023-03-27 04:58:42,792 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:42,904 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1387 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/ce3aa0157b7b46c49c19d2223e1add58 2023-03-27 04:58:42,908 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/d70fbc41bdb44fdda644cc6826a2fa12 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/d70fbc41bdb44fdda644cc6826a2fa12 2023-03-27 04:58:42,912 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/d70fbc41bdb44fdda644cc6826a2fa12, entries=150, sequenceid=1387, filesize=25.4 K 2023-03-27 04:58:42,913 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/6bf05e3e05554519bde8f5e792fce413 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/6bf05e3e05554519bde8f5e792fce413 2023-03-27 04:58:42,917 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/6bf05e3e05554519bde8f5e792fce413, entries=150, sequenceid=1387, filesize=11.9 K 2023-03-27 04:58:42,918 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/ce3aa0157b7b46c49c19d2223e1add58 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ce3aa0157b7b46c49c19d2223e1add58 2023-03-27 04:58:42,936 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ce3aa0157b7b46c49c19d2223e1add58, entries=150, sequenceid=1387, filesize=11.9 K 2023-03-27 04:58:42,937 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for a34144f135b39d6005ad383002fabe3a in 648ms, sequenceid=1387, compaction requested=false 2023-03-27 04:58:42,937 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:42,937 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:42,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:42,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:42,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:42,937 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:42,938 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,938 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:42,939 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:42,939 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:42,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:42,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19bb9242[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:42,940 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:42,940 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,941 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,941 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:42,941 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:42,941 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:42,941 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,941 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:42,942 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,942 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:42,942 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:42,942 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:42,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:42,945 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:42,945 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:42,945 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:42,945 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:42,945 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:42,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:42,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:42,948 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:42,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:42,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:42,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:42,949 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:42,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:42,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:42,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:42,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:42,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:42,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:42,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:42,952 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:42,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:42,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:42,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:42,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:42,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:42,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:42,952 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:42,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:42,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:42,952 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:43,053 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:43,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:43,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:43,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:43,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:43,062 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:43,062 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:43,062 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:43,062 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:43,063 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,063 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,063 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,063 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:43,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:43,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:43,063 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,063 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:43,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:43,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:43,065 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:43,065 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:43,065 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:43,065 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:43,065 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:43,065 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,065 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:43,065 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:43,065 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,065 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:43,065 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:43,065 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,065 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:43,065 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:43,067 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,067 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,069 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,070 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:43,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@130f5f29[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:43,072 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:43,072 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,073 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,073 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:43,074 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:43,074 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:43,074 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,074 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:43,075 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:43,075 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:43,075 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:43,075 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:43,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:43,077 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:43,077 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:43,077 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:43,077 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:43,077 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:43,078 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,078 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:43,078 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,079 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,081 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,083 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:43,083 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:43,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:43,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:43,083 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:43,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:43,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:43,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:43,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:43,083 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:43,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:43,083 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:43,086 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,086 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,086 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,086 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,086 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,086 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,086 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:43,110 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:58:43,110 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:43,110 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:43,110 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:43,110 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:43,110 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:43,110 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:43,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893123130, max timestamp=1679893123140Num uniques 859; 2023-03-27 04:58:43,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:58:43,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893123130, max timestamp=1679893123140Num uniques 859; 2023-03-27 04:58:43,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:58:43,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893123130, max timestamp=1679893123140Num uniques 859; 2023-03-27 04:58:43,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:58:43,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:43,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893183161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:43,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893183161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:43,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893183164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:43,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893183165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:43,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50916 deadline: 1679893183169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,184 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:43,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:43,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:43,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:43,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:43,204 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:43,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:43,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:43,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:43,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,204 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,205 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,205 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:43,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:43,205 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,205 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:43,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:43,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:43,206 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:43,206 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:43,206 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:43,206 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:43,206 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:43,206 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,207 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:43,207 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:43,207 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,207 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:43,207 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:43,207 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,207 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:43,207 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:43,208 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,208 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,208 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,208 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:43,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ebb16bd[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:43,209 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:43,210 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,210 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,210 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:43,210 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:43,210 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,210 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:43,210 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:43,212 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:43,212 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,212 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,212 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:43,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:43,215 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:43,215 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:43,215 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:43,215 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:43,215 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:43,216 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,216 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,216 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:43,216 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,217 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,219 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,219 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,219 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,219 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:43,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:43,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:43,219 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:43,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:43,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:43,219 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,220 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,220 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,220 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,219 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:43,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:43,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:43,220 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:43,220 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:43,240 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/ceb367ec19f54f279db60e6c0033db17 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/ceb367ec19f54f279db60e6c0033db17 2023-03-27 04:58:43,244 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a34144f135b39d6005ad383002fabe3a/A of a34144f135b39d6005ad383002fabe3a into ceb367ec19f54f279db60e6c0033db17(size=26.0 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:43,244 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:43,244 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/A, priority=11, startTime=1679893122255; duration=0sec 2023-03-27 04:58:43,244 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:43,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:43,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893183266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:43,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893183266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:43,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893183268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:43,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893183271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,320 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:43,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:43,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:43,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:43,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:43,323 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:43,323 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:43,323 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:43,323 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:43,324 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,324 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,325 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:43,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:43,325 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,325 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,325 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:43,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:43,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:43,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:43,326 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:43,326 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:43,326 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:43,334 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,334 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:43,334 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:43,334 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:43,339 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,339 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,340 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,340 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:43,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20fa4c72[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:43,343 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:43,343 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,344 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,344 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:43,344 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:43,344 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:43,344 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,344 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:43,345 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:43,345 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,345 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:43,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,345 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:43,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:43,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:43,347 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:43,347 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:43,347 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:43,347 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:43,347 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:43,348 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,348 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,348 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:43,349 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,350 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,351 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,352 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,352 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:43,352 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:43,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:43,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:43,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:43,352 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:43,352 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,352 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,352 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,352 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:43,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:43,352 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:43,352 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,352 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:43,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:43,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:43,452 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:43,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:43,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:43,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:43,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:43,455 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:43,455 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:43,455 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:43,455 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:43,455 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,455 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,456 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,456 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:43,456 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:43,456 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:43,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:43,457 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:43,457 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:43,457 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:43,457 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:43,457 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:43,457 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,457 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:43,457 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:43,457 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,457 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:43,458 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:43,458 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,458 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:43,458 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:43,458 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,458 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,459 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,459 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:43,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6596f3ad[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:43,460 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:43,460 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,461 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:43,461 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:43,461 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:43,461 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,461 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:43,461 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,461 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,462 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:43,462 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:43,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:43,464 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:43,464 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:43,464 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:43,464 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:43,464 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:43,465 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,465 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:43,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,465 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,466 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,467 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,468 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,468 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,468 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,468 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,468 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,468 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,468 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,468 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:43,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:43,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:43,468 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:43,468 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:43,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:43,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:43,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:43,469 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:43,469 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:43,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:43,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:43,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:43,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893183469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:43,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893183470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:43,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893183472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:43,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893183475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,520 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327b6946245b2b94664a2d83bf914635800 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327b6946245b2b94664a2d83bf914635800 2023-03-27 04:58:43,524 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1400, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/66c1cbec50fb4587bdb1af7f9fba2af2 2023-03-27 04:58:43,534 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1400 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/ac55100f6b854014bee9fd926957a263 2023-03-27 04:58:43,569 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:43,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:43,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:43,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:43,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:43,572 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:43,572 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:43,572 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:43,572 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:43,572 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,572 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,573 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,573 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:43,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:43,573 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,573 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:43,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:43,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:43,575 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:43,575 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:43,575 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,575 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:43,575 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:43,575 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,575 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:43,575 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:43,575 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,575 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:43,575 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:43,576 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,576 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,576 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,576 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:43,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c890e21[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:43,578 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:43,578 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,578 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:43,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:43,578 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:43,579 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,579 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:43,579 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,579 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:43,579 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:43,579 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:43,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:43,581 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:43,581 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:43,581 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:43,581 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:43,582 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:43,582 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,582 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:43,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,583 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:43,583 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,584 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,586 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,586 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:43,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:43,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:43,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:43,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:43,587 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:43,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:43,587 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:43,587 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:43,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:43,687 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:43,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:43,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:43,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:43,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:43,691 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:43,692 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:43,692 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:43,692 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:43,692 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,692 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,693 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:43,693 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:43,693 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,693 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:43,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:43,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:43,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:43,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:43,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:43,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:43,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:43,694 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:43,694 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,694 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:43,694 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:43,694 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,695 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:43,695 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:43,695 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,695 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:43,695 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:43,696 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,696 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,697 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,697 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:43,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a01fff3[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:43,700 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:43,700 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,706 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,706 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:43,706 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:43,706 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:43,706 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,706 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:43,707 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,707 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:43,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,707 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:43,707 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:43,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:43,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:43,710 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:43,710 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:43,710 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:43,710 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:43,710 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:43,711 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,712 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:43,712 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,712 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,714 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,717 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:43,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:43,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:43,717 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:43,717 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:43,718 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,718 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,718 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,718 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,718 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,718 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,718 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,722 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:43,722 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:43,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:43,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:43,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:43,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:43,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893183773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:43,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893183773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:43,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893183775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:43,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893183781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,817 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:43,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:43,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:43,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:43,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:43,820 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:43,820 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:43,820 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:43,820 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:43,821 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,821 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,821 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,821 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:43,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:43,822 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,822 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:43,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:43,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:43,823 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,823 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:43,823 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:43,823 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:43,824 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:43,824 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,824 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:43,824 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:43,825 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,825 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,825 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,825 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:43,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78eaab1d[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:43,827 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:43,827 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,828 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,828 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:43,828 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:43,828 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:43,828 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,828 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:43,829 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:43,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:43,829 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:43,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:43,832 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:43,832 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:43,832 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:43,832 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:43,832 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:43,833 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,833 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:43,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,833 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,834 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,835 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,837 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:43,837 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:43,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:43,837 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,837 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:43,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:43,837 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:43,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:43,837 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:43,837 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,837 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,837 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,838 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,838 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:43,838 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:43,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:43,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:43,938 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:43,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:43,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:43,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:43,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:43,950 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:43,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:43,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:43,950 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:43,950 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1400 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/10d248e9144c4c9e9bef5544ff075855 2023-03-27 04:58:43,951 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,951 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,952 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:43,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:43,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:43,952 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,952 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,952 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:43,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:43,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:43,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:43,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:43,954 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:43,954 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:43,954 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:43,955 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/66c1cbec50fb4587bdb1af7f9fba2af2 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/66c1cbec50fb4587bdb1af7f9fba2af2 2023-03-27 04:58:43,959 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/66c1cbec50fb4587bdb1af7f9fba2af2, entries=350, sequenceid=1400, filesize=52.9 K 2023-03-27 04:58:43,960 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/ac55100f6b854014bee9fd926957a263 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/ac55100f6b854014bee9fd926957a263 2023-03-27 04:58:43,963 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/ac55100f6b854014bee9fd926957a263, entries=150, sequenceid=1400, filesize=11.9 K 2023-03-27 04:58:43,964 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/10d248e9144c4c9e9bef5544ff075855 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/10d248e9144c4c9e9bef5544ff075855 2023-03-27 04:58:43,970 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:43,970 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:43,970 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,970 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:43,970 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:43,970 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,970 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:43,970 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:43,970 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,970 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:43,970 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:43,971 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/10d248e9144c4c9e9bef5544ff075855, entries=150, sequenceid=1400, filesize=11.9 K 2023-03-27 04:58:43,971 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,971 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,972 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:43,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,973 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a34144f135b39d6005ad383002fabe3a in 863ms, sequenceid=1400, compaction requested=true 2023-03-27 04:58:43,973 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:43,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,973 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:43,973 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:43,973 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:43,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:43,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ab32436[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:43,973 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:43,974 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:43,974 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:43,974 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,975 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,975 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 106920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:43,975 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/A is initiating minor compaction (all files) 2023-03-27 04:58:43,975 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/A in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,975 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/ceb367ec19f54f279db60e6c0033db17, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/d70fbc41bdb44fdda644cc6826a2fa12, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/66c1cbec50fb4587bdb1af7f9fba2af2] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=104.4 K 2023-03-27 04:58:43,976 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:43,976 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/B is initiating minor compaction (all files) 2023-03-27 04:58:43,976 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:43,976 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/B in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:43,976 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/518e6513e7d443e1a7bcc154e3a50347, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/6bf05e3e05554519bde8f5e792fce413, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/ac55100f6b854014bee9fd926957a263] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=36.3 K 2023-03-27 04:58:43,976 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:43,977 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:43,976 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,977 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:43,977 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting ceb367ec19f54f279db60e6c0033db17, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=1351, earliestPutTs=1679893120961 2023-03-27 04:58:43,978 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 518e6513e7d443e1a7bcc154e3a50347, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1351, earliestPutTs=1679893120961 2023-03-27 04:58:43,978 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting d70fbc41bdb44fdda644cc6826a2fa12, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1387, earliestPutTs=1679893121031 2023-03-27 04:58:43,978 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,978 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,978 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 66c1cbec50fb4587bdb1af7f9fba2af2, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=1400, earliestPutTs=1679893123074 2023-03-27 04:58:43,978 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:43,978 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,978 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 6bf05e3e05554519bde8f5e792fce413, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1387, earliestPutTs=1679893121031 2023-03-27 04:58:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:43,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:43,981 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:43,981 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:43,981 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:43,981 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:43,981 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:43,982 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting ac55100f6b854014bee9fd926957a263, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1400, earliestPutTs=1679893123103 2023-03-27 04:58:43,983 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,983 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,983 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:43,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,984 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,985 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,988 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,988 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,988 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:43,988 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:43,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:43,989 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:43,989 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:43,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:43,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:43,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:43,989 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:43,989 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,989 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:43,989 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:43,989 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:43,989 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:43,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:43,991 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#B#compaction#528 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:44,006 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#A#compaction#527 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:44,026 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/38696156fc994a63a139911b7cb11f85 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/38696156fc994a63a139911b7cb11f85 2023-03-27 04:58:44,032 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a34144f135b39d6005ad383002fabe3a/B of a34144f135b39d6005ad383002fabe3a into 38696156fc994a63a139911b7cb11f85(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:44,032 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:44,032 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/B, priority=13, startTime=1679893123973; duration=0sec 2023-03-27 04:58:44,032 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:44,032 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-27 04:58:44,033 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-27 04:58:44,033 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/C is initiating minor compaction (all files) 2023-03-27 04:58:44,033 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/C in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:44,033 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/5764f3f6e6334627ba7b6b74dc063910, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ce3aa0157b7b46c49c19d2223e1add58, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/10d248e9144c4c9e9bef5544ff075855] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=36.3 K 2023-03-27 04:58:44,033 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 5764f3f6e6334627ba7b6b74dc063910, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1351, earliestPutTs=1679893120961 2023-03-27 04:58:44,034 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting ce3aa0157b7b46c49c19d2223e1add58, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1387, earliestPutTs=1679893121031 2023-03-27 04:58:44,034 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 10d248e9144c4c9e9bef5544ff075855, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1400, earliestPutTs=1679893123103 2023-03-27 04:58:44,041 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/71f9243e80fd47948eda363f8e1b726b as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/71f9243e80fd47948eda363f8e1b726b 2023-03-27 04:58:44,043 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#C#compaction#529 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:44,047 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a34144f135b39d6005ad383002fabe3a/A of a34144f135b39d6005ad383002fabe3a into 71f9243e80fd47948eda363f8e1b726b(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:44,048 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:44,048 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/A, priority=13, startTime=1679893123973; duration=0sec 2023-03-27 04:58:44,048 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:44,089 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:44,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:44,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:44,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:44,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:44,092 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:44,092 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:44,092 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:44,092 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:44,092 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:44,092 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:44,093 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:44,093 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,093 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,093 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:44,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:44,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:44,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:44,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:44,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:44,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:44,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:44,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:44,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:44,095 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:44,095 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:44,095 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:44,098 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:44,098 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:44,098 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:44,098 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:44,098 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-27 04:58:44,098 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:44,099 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:44,099 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:44,099 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:44,099 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:44,099 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-27 04:58:44,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:44,279 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:44,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:44,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893184287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:44,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893184287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:44,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893184288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:44,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893184290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:44,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893184391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:44,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893184391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:44,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893184392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:44,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893184395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,462 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/126c63a7929543fc9ffdaeeb71ec6ca1 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/126c63a7929543fc9ffdaeeb71ec6ca1 2023-03-27 04:58:44,475 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a34144f135b39d6005ad383002fabe3a/C of a34144f135b39d6005ad383002fabe3a into 126c63a7929543fc9ffdaeeb71ec6ca1(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:44,475 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:44,475 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/C, priority=13, startTime=1679893123974; duration=0sec 2023-03-27 04:58:44,475 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:44,526 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032728c203fdb9014483ae9de126174e5bb9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032728c203fdb9014483ae9de126174e5bb9 2023-03-27 04:58:44,550 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1440, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/7ec88bf754ce429d9e69ed6d989155b9 2023-03-27 04:58:44,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:44,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:44,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893184599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893184599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:44,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893184599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,600 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:44,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893184599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,600 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1440 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/eb50cd60f6c344f8a365a8b55439e0eb 2023-03-27 04:58:44,624 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1440 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/1cb93a5dc1fc483a907b06e51a8d98a4 2023-03-27 04:58:44,627 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/7ec88bf754ce429d9e69ed6d989155b9 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/7ec88bf754ce429d9e69ed6d989155b9 2023-03-27 04:58:44,630 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/7ec88bf754ce429d9e69ed6d989155b9, entries=150, sequenceid=1440, filesize=25.4 K 2023-03-27 04:58:44,631 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/eb50cd60f6c344f8a365a8b55439e0eb as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/eb50cd60f6c344f8a365a8b55439e0eb 2023-03-27 04:58:44,634 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/eb50cd60f6c344f8a365a8b55439e0eb, entries=150, sequenceid=1440, filesize=11.9 K 2023-03-27 04:58:44,635 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/1cb93a5dc1fc483a907b06e51a8d98a4 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/1cb93a5dc1fc483a907b06e51a8d98a4 2023-03-27 04:58:44,638 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/1cb93a5dc1fc483a907b06e51a8d98a4, entries=150, sequenceid=1440, filesize=11.9 K 2023-03-27 04:58:44,639 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for a34144f135b39d6005ad383002fabe3a in 541ms, sequenceid=1440, compaction requested=false 2023-03-27 04:58:44,639 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:44,639 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:44,639 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:44,639 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:44,639 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,639 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:44,639 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:44,640 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:44,640 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:44,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:44,641 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:44,641 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:44,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:44,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:44,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:44,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:44,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:44,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e0fcf47[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:44,642 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:44,642 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:44,643 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:44,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:44,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:44,643 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:44,643 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:44,643 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:44,643 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,644 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:44,644 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:44,644 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:44,644 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:44,645 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:44,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:44,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:44,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:44,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:44,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:44,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:44,648 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:44,648 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:44,648 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:44,648 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:44,648 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:44,649 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:44,649 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:44,650 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:44,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:44,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:44,650 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,652 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,655 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:44,655 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:44,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:44,655 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:44,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:44,655 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:44,655 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:44,655 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:44,655 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:44,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:44,655 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:44,655 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,656 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:44,656 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:44,656 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:44,756 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:44,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:44,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:44,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:44,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:44,759 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:44,759 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:44,759 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:44,759 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:44,759 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:44,759 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:44,760 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:44,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:44,760 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:44,760 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:44,760 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:44,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:44,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:44,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:44,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:44,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:44,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:44,774 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:44,774 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:44,774 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:44,775 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:44,775 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:44,775 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:44,776 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:44,776 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:58:44,776 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:44,778 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:44,778 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:44,778 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:44,778 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:44,778 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:44,818 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327a03b2123800d42b6be6c4ebe5bf26c27 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327a03b2123800d42b6be6c4ebe5bf26c27 2023-03-27 04:58:44,826 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1451, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/ef4991d24fff4003afea9d49c34a3255 2023-03-27 04:58:44,902 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:44,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:44,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893124958, max timestamp=1679893124995Num uniques 859; 2023-03-27 04:58:44,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893124958, max timestamp=1679893124995Num uniques 859; 2023-03-27 04:58:44,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:58:44,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:58:44,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679893124958, max timestamp=1679893124995Num uniques 859; 2023-03-27 04:58:44,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-27 04:58:45,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893185018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893185019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893185034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893185035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893185136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893185136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893185145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893185151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50916 deadline: 1679893185188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,214 DEBUG [Thread-3617] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., hostname=jenkins-hbase20.apache.org,44923,1679892955505, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-27 04:58:45,261 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1451 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/c0cc737d52994966acbbed39e290af77 2023-03-27 04:58:45,297 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1451 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/8e5118ac85164c7ab657e6bb4f46b9e4 2023-03-27 04:58:45,309 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/ef4991d24fff4003afea9d49c34a3255 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/ef4991d24fff4003afea9d49c34a3255 2023-03-27 04:58:45,313 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/ef4991d24fff4003afea9d49c34a3255, entries=150, sequenceid=1451, filesize=25.4 K 2023-03-27 04:58:45,314 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/c0cc737d52994966acbbed39e290af77 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/c0cc737d52994966acbbed39e290af77 2023-03-27 04:58:45,334 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/c0cc737d52994966acbbed39e290af77, entries=150, sequenceid=1451, filesize=11.9 K 2023-03-27 04:58:45,335 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/8e5118ac85164c7ab657e6bb4f46b9e4 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/8e5118ac85164c7ab657e6bb4f46b9e4 2023-03-27 04:58:45,343 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/8e5118ac85164c7ab657e6bb4f46b9e4, entries=150, sequenceid=1451, filesize=11.9 K 2023-03-27 04:58:45,344 INFO [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a34144f135b39d6005ad383002fabe3a in 568ms, sequenceid=1451, compaction requested=true 2023-03-27 04:58:45,344 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:45,344 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:45,344 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:45,344 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:45,344 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:45,345 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-03-27 04:58:45,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:45,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-27 04:58:45,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:45,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-27 04:58:45,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:45,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-27 04:58:45,347 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:45,347 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:45,348 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,348 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:45,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:45,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:45,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:45,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:45,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:45,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:45,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b744b12[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:45,350 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:45,350 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,350 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,350 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:45,351 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,351 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:45,351 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:45,351 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:45,351 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:45,354 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:45,355 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:45,354 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:45,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:45,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:45,355 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:45,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:45,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:45,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:45,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:45,357 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:45,358 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:45,358 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:45,358 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:45,358 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:45,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893185358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893185360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893185361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,364 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,364 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:45,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,365 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:45,366 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:45,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893185363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,368 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:45,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,373 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:45,373 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:45,373 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,373 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:45,373 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:45,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:45,373 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:45,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:45,373 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:45,374 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:45,374 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:45,373 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,374 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,374 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:45,407 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032726b3714d8d984f9c89faab4929e61a19 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032726b3714d8d984f9c89faab4929e61a19 2023-03-27 04:58:45,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893185465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893185465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44923] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893185465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893185469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,474 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:45,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:45,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:45,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:45,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:45,478 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:45,478 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:45,478 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:45,478 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:45,478 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,478 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:45,479 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:45,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:45,479 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:45,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:45,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:45,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:45,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:45,481 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,481 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:45,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-27 04:58:45,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:45,486 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:45,486 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:45,486 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:45,494 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:45,494 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:45,494 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:45,494 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:45,494 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:45,494 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:45,494 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:45,495 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:45,495 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,495 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:45,495 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:45,496 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:45,498 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:45,499 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,499 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:45,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:45,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@466bb722[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:45,500 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:45,500 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,501 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,501 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:45,501 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:45,501 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:45,502 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:45,502 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:45,502 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:45,502 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,503 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:45,503 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:45,503 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:45,503 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:45,503 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:45,503 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:45,503 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:45,504 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,504 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:45,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,504 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:45,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:45,506 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:45,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,507 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,514 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:45,515 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:45,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:45,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:45,515 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,515 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:45,515 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:45,515 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,515 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,515 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,515 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,515 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:45,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:45,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:45,515 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:45,515 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:45,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:45,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-27 04:58:45,615 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:45,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:45,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-27 04:58:45,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-27 04:58:45,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-27 04:58:45,619 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-27 04:58:45,619 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:45,619 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-27 04:58:45,619 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-27 04:58:45,620 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,620 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:45,621 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,622 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:45,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:45,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:45,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,622 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-27 04:58:45,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:45,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-27 04:58:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:45,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-27 04:58:45,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-27 04:58:45,626 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-27 04:58:45,626 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-27 04:58:45,626 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-27 04:58:45,635 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-27 04:58:45,635 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-27 04:58:45,636 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:45,636 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. started... 2023-03-27 04:58:45,636 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. as already flushing 2023-03-27 04:58:45,636 DEBUG [rs(jenkins-hbase20.apache.org,44923,1679892955505)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:45,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-27 04:58:45,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-27 04:58:45,636 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,637 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-27 04:58:45,637 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-27 04:58:45,638 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:45,640 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,640 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,640 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-27 04:58:45,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,44923,1679892955505' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-27 04:58:45,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75caa94d[Count = 0] remaining members to acquire global barrier 2023-03-27 04:58:45,641 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-27 04:58:45,641 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,641 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,642 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-27 04:58:45,642 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-27 04:58:45,642 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,44923,1679892955505' in zk 2023-03-27 04:58:45,642 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,642 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-27 04:58:45,644 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:45,645 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-27 04:58:45,645 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:45,645 DEBUG [member: 'jenkins-hbase20.apache.org,44923,1679892955505' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-27 04:58:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:45,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:45,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:45,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:45,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,44923,1679892955505': 2023-03-27 04:58:45,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,44923,1679892955505' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-27 04:58:45,647 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-27 04:58:45,647 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-27 04:58:45,647 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-27 04:58:45,648 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-27 04:58:45,648 INFO [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-27 04:58:45,649 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,649 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:45,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,651 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-27 04:58:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-27 04:58:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-27 04:58:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:45,652 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-27 04:58:45,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-27 04:58:45,654 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-27 04:58:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-27 04:58:45,665 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-27 04:58:45,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-27 04:58:45,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-27 04:58:45,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-27 04:58:45,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-27 04:58:45,666 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,666 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:45,666 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-27 04:58:45,666 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,666 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,666 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-27 04:58:45,666 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-27 04:58:45,666 DEBUG [(jenkins-hbase20.apache.org,36293,1679892954567)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-27 04:58:45,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-27 04:58:45,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-27 04:58:45,666 DEBUG [Thread-3623] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-27 04:58:45,666 DEBUG [Thread-3623] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-27 04:58:45,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50936 deadline: 1679893185668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50926 deadline: 1679893185669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50974 deadline: 1679893185669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-27 04:58:45,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44923] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:50896 deadline: 1679893185673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:45,720 DEBUG [Thread-3630] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2daeb795 to 127.0.0.1:61059 2023-03-27 04:58:45,720 DEBUG [Thread-3630] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:45,721 DEBUG [Thread-3628] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x070ab5f5 to 127.0.0.1:61059 2023-03-27 04:58:45,721 DEBUG [Thread-3628] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:45,722 DEBUG [Thread-3624] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x078cd331 to 127.0.0.1:61059 2023-03-27 04:58:45,722 DEBUG [Thread-3624] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:45,723 DEBUG [Thread-3632] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28ac2b09 to 127.0.0.1:61059 2023-03-27 04:58:45,723 DEBUG [Thread-3632] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:45,725 DEBUG [Thread-3626] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d74fc48 to 127.0.0.1:61059 2023-03-27 04:58:45,725 DEBUG [Thread-3626] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:45,766 DEBUG [Thread-3623] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-27 04:58:45,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-27 04:58:45,840 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1489, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/48b034b9cb98439ea629db2d3483db74 2023-03-27 04:58:45,848 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1489 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/553d6848e7344b64b2505c4e513bca40 2023-03-27 04:58:45,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1489 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/8d4bbdde4f3b4bc88ff08dc9c02a6900 2023-03-27 04:58:45,860 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/48b034b9cb98439ea629db2d3483db74 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/48b034b9cb98439ea629db2d3483db74 2023-03-27 04:58:45,863 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/48b034b9cb98439ea629db2d3483db74, entries=200, sequenceid=1489, filesize=32.3 K 2023-03-27 04:58:45,864 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/553d6848e7344b64b2505c4e513bca40 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/553d6848e7344b64b2505c4e513bca40 2023-03-27 04:58:45,867 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/553d6848e7344b64b2505c4e513bca40, entries=150, sequenceid=1489, filesize=11.9 K 2023-03-27 04:58:45,868 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/8d4bbdde4f3b4bc88ff08dc9c02a6900 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/8d4bbdde4f3b4bc88ff08dc9c02a6900 2023-03-27 04:58:45,871 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/8d4bbdde4f3b4bc88ff08dc9c02a6900, entries=150, sequenceid=1489, filesize=11.9 K 2023-03-27 04:58:45,872 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for a34144f135b39d6005ad383002fabe3a in 527ms, sequenceid=1489, compaction requested=true 2023-03-27 04:58:45,872 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:45,872 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:45,872 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:58:45,872 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:58:45,872 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:45,872 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:45,873 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111940 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:58:45,873 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/A is initiating minor compaction (all files) 2023-03-27 04:58:45,873 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:58:45,873 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/A in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:45,873 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/B is initiating minor compaction (all files) 2023-03-27 04:58:45,873 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/71f9243e80fd47948eda363f8e1b726b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/7ec88bf754ce429d9e69ed6d989155b9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/ef4991d24fff4003afea9d49c34a3255, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/48b034b9cb98439ea629db2d3483db74] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=109.3 K 2023-03-27 04:58:45,873 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/B in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:45,873 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/38696156fc994a63a139911b7cb11f85, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/eb50cd60f6c344f8a365a8b55439e0eb, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/c0cc737d52994966acbbed39e290af77, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/553d6848e7344b64b2505c4e513bca40] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=48.3 K 2023-03-27 04:58:45,873 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 71f9243e80fd47948eda363f8e1b726b, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=1400, earliestPutTs=1679893123103 2023-03-27 04:58:45,874 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 38696156fc994a63a139911b7cb11f85, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=1400, earliestPutTs=1679893123103 2023-03-27 04:58:45,874 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 7ec88bf754ce429d9e69ed6d989155b9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1440, earliestPutTs=1679893123150 2023-03-27 04:58:45,874 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting eb50cd60f6c344f8a365a8b55439e0eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1440, earliestPutTs=1679893123150 2023-03-27 04:58:45,874 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting ef4991d24fff4003afea9d49c34a3255, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1451, earliestPutTs=1679893124279 2023-03-27 04:58:45,874 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting c0cc737d52994966acbbed39e290af77, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1451, earliestPutTs=1679893124279 2023-03-27 04:58:45,874 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] compactions.Compactor(207): Compacting 48b034b9cb98439ea629db2d3483db74, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1489, earliestPutTs=1679893125011 2023-03-27 04:58:45,875 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 553d6848e7344b64b2505c4e513bca40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1489, earliestPutTs=1679893125011 2023-03-27 04:58:45,882 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#B#compaction#539 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:45,888 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#A#compaction#540 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:45,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44923] regionserver.HRegion(9158): Flush requested on a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:45,975 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-27 04:58:45,975 DEBUG [Thread-3619] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08d3d8bc to 127.0.0.1:61059 2023-03-27 04:58:45,975 DEBUG [Thread-3613] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50131d14 to 127.0.0.1:61059 2023-03-27 04:58:45,975 DEBUG [Thread-3619] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:45,975 DEBUG [Thread-3613] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:45,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:45,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:45,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:45,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:45,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:45,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:45,977 DEBUG [Thread-3621] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e6ad437 to 127.0.0.1:61059 2023-03-27 04:58:45,977 DEBUG [Thread-3621] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:45,983 DEBUG [Thread-3615] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6722b207 to 127.0.0.1:61059 2023-03-27 04:58:45,983 DEBUG [Thread-3615] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:46,299 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/98b44bb4e37342d79ece245f9c783b11 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/98b44bb4e37342d79ece245f9c783b11 2023-03-27 04:58:46,302 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a34144f135b39d6005ad383002fabe3a/B of a34144f135b39d6005ad383002fabe3a into 98b44bb4e37342d79ece245f9c783b11(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:46,302 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:46,302 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/B, priority=12, startTime=1679893125872; duration=0sec 2023-03-27 04:58:46,302 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-27 04:58:46,302 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-27 04:58:46,303 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-27 04:58:46,303 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1912): a34144f135b39d6005ad383002fabe3a/C is initiating minor compaction (all files) 2023-03-27 04:58:46,303 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a34144f135b39d6005ad383002fabe3a/C in TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:46,303 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/126c63a7929543fc9ffdaeeb71ec6ca1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/1cb93a5dc1fc483a907b06e51a8d98a4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/8e5118ac85164c7ab657e6bb4f46b9e4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/8d4bbdde4f3b4bc88ff08dc9c02a6900] into tmpdir=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp, totalSize=48.3 K 2023-03-27 04:58:46,303 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 126c63a7929543fc9ffdaeeb71ec6ca1, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=1400, earliestPutTs=1679893123103 2023-03-27 04:58:46,303 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 1cb93a5dc1fc483a907b06e51a8d98a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1440, earliestPutTs=1679893123150 2023-03-27 04:58:46,304 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 8e5118ac85164c7ab657e6bb4f46b9e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1451, earliestPutTs=1679893124279 2023-03-27 04:58:46,304 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] compactions.Compactor(207): Compacting 8d4bbdde4f3b4bc88ff08dc9c02a6900, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1489, earliestPutTs=1679893125011 2023-03-27 04:58:46,308 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] throttle.PressureAwareThroughputController(145): a34144f135b39d6005ad383002fabe3a#C#compaction#542 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-27 04:58:46,315 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/25ac5d6a0966478995a387adde1baf78 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/25ac5d6a0966478995a387adde1baf78 2023-03-27 04:58:46,319 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a34144f135b39d6005ad383002fabe3a/C of a34144f135b39d6005ad383002fabe3a into 25ac5d6a0966478995a387adde1baf78(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-27 04:58:46,319 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:46,319 INFO [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/C, priority=12, startTime=1679893125872; duration=0sec 2023-03-27 04:58:46,319 DEBUG [RS:0;jenkins-hbase20:44923-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:46,395 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327b61b16fb3d8744dfa02208ae982b633e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327b61b16fb3d8744dfa02208ae982b633e 2023-03-27 04:58:46,801 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1501, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/a1e07bb4296e41e689fea876a38a738d 2023-03-27 04:58:47,122 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/5361bdfd943c4d689379319084db40d1 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/5361bdfd943c4d689379319084db40d1 2023-03-27 04:58:47,127 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a34144f135b39d6005ad383002fabe3a/A of a34144f135b39d6005ad383002fabe3a into 5361bdfd943c4d689379319084db40d1(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-27 04:58:47,127 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:47,127 INFO [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a., storeName=a34144f135b39d6005ad383002fabe3a/A, priority=12, startTime=1679893125872; duration=1sec 2023-03-27 04:58:47,128 DEBUG [RS:0;jenkins-hbase20:44923-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-27 04:58:47,218 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1501 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/c5e584dcc0e9470d931e7f9b9193e685 2023-03-27 04:58:47,633 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1501 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/0602f8348e254cecac36ea6e5d8a869f 2023-03-27 04:58:47,636 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/a1e07bb4296e41e689fea876a38a738d as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/a1e07bb4296e41e689fea876a38a738d 2023-03-27 04:58:47,639 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/a1e07bb4296e41e689fea876a38a738d, entries=150, sequenceid=1501, filesize=25.4 K 2023-03-27 04:58:47,639 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/c5e584dcc0e9470d931e7f9b9193e685 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/c5e584dcc0e9470d931e7f9b9193e685 2023-03-27 04:58:47,642 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/c5e584dcc0e9470d931e7f9b9193e685, entries=150, sequenceid=1501, filesize=11.9 K 2023-03-27 04:58:47,643 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/0602f8348e254cecac36ea6e5d8a869f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/0602f8348e254cecac36ea6e5d8a869f 2023-03-27 04:58:47,645 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/0602f8348e254cecac36ea6e5d8a869f, entries=150, sequenceid=1501, filesize=11.9 K 2023-03-27 04:58:47,646 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=6.71 KB/6870 for a34144f135b39d6005ad383002fabe3a in 1671ms, sequenceid=1501, compaction requested=false 2023-03-27 04:58:47,646 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:49,250 DEBUG [Thread-3617] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x48502121 to 127.0.0.1:61059 2023-03-27 04:58:49,250 DEBUG [Thread-3617] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:49,250 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-27 04:58:49,250 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 289 2023-03-27 04:58:49,250 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 287 2023-03-27 04:58:49,250 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 269 2023-03-27 04:58:49,250 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 286 2023-03-27 04:58:49,251 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(394): wrote 281 2023-03-27 04:58:49,251 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-27 04:58:49,251 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(398): read 4152 2023-03-27 04:58:49,251 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(398): read 4120 2023-03-27 04:58:49,251 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(398): read 4048 2023-03-27 04:58:49,251 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(398): read 4139 2023-03-27 04:58:49,251 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(398): read 4121 2023-03-27 04:58:49,251 INFO [Listener at localhost.localdomain/43311] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-27 04:58:49,251 INFO [Listener at localhost.localdomain/43311] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-27 04:58:49,251 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b200ba7 to 127.0.0.1:61059 2023-03-27 04:58:49,251 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:49,251 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-27 04:58:49,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-27 04:58:49,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-27 04:58:49,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-27 04:58:49,256 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893129256"}]},"ts":"1679893129256"} 2023-03-27 04:58:49,257 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-27 04:58:49,258 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-27 04:58:49,259 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a34144f135b39d6005ad383002fabe3a, UNASSIGN}] 2023-03-27 04:58:49,261 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a34144f135b39d6005ad383002fabe3a, UNASSIGN 2023-03-27 04:58:49,266 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=a34144f135b39d6005ad383002fabe3a, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:49,266 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893129266"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679893129266"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679893129266"}]},"ts":"1679893129266"} 2023-03-27 04:58:49,267 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505}] 2023-03-27 04:58:49,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-27 04:58:49,419 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:49,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing a34144f135b39d6005ad383002fabe3a, disabling compactions & flushes 2023-03-27 04:58:49,419 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:49,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:49,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. after waiting 0 ms 2023-03-27 04:58:49,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:49,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing a34144f135b39d6005ad383002fabe3a 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-27 04:58:49,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=A 2023-03-27 04:58:49,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:49,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=B 2023-03-27 04:58:49,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:49,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a34144f135b39d6005ad383002fabe3a, store=C 2023-03-27 04:58:49,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-27 04:58:49,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-27 04:58:49,833 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230327d0de25cf1c0844fb841837f0e5c42923 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327d0de25cf1c0844fb841837f0e5c42923 2023-03-27 04:58:49,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-27 04:58:50,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1509, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/465c6691746947098b3800a426cf9459 2023-03-27 04:58:50,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-27 04:58:50,648 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=1509 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/d07216f6e728456caabd939599e25e0f 2023-03-27 04:58:51,068 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=1509 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/124fe40574fb495abdff5d45733e9fdd 2023-03-27 04:58:51,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/A/465c6691746947098b3800a426cf9459 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/465c6691746947098b3800a426cf9459 2023-03-27 04:58:51,078 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/465c6691746947098b3800a426cf9459, entries=100, sequenceid=1509, filesize=18.6 K 2023-03-27 04:58:51,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/B/d07216f6e728456caabd939599e25e0f as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/d07216f6e728456caabd939599e25e0f 2023-03-27 04:58:51,085 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/d07216f6e728456caabd939599e25e0f, entries=100, sequenceid=1509, filesize=9.5 K 2023-03-27 04:58:51,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/.tmp/C/124fe40574fb495abdff5d45733e9fdd as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/124fe40574fb495abdff5d45733e9fdd 2023-03-27 04:58:51,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/124fe40574fb495abdff5d45733e9fdd, entries=100, sequenceid=1509, filesize=9.5 K 2023-03-27 04:58:51,092 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for a34144f135b39d6005ad383002fabe3a in 1672ms, sequenceid=1509, compaction requested=true 2023-03-27 04:58:51,132 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/2151ffec0fe440f2acb7fedb6033c90a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/ef03f4089e9946a5891f9408aa2ae999, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/283e844b2d1c4cd99de1fd67592124cc, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/7b8b2d649b8a49f6aedc8cd948e1e531, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/c1d2d6cd788c472591d611811df08ef7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/fcbe22d7477f490dba0efbe26e76f4d4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/86be74830c85442dbda54c9fde901971, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/642d532ee37345bb8a64a3102af6af1d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/c798d48c7dcf4d55a90620c4cf27ee1e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/3b57dcd96b2b4dbfb3e35c5b42dc7bc3, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/51fe687a6eae4ddbbaf2168dd462ebb5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/e56de8e34969424bbdb26538d85fdb38, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/750129795cd24615922209900dcf1c74, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/5af325f076ba4cf7a776db0609c314c6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/441dd5c3b844405394d811634358c4b0, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/f2e6edd98aba46a1bae0bc55e2b332a4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/bbca2454412a4cc19b82dce057cdf9e3, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/8c4347c1f03e47b4a4710db33e0cda30, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/ceb367ec19f54f279db60e6c0033db17, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/d70fbc41bdb44fdda644cc6826a2fa12, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/66c1cbec50fb4587bdb1af7f9fba2af2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/71f9243e80fd47948eda363f8e1b726b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/7ec88bf754ce429d9e69ed6d989155b9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/ef4991d24fff4003afea9d49c34a3255, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/48b034b9cb98439ea629db2d3483db74] to archive 2023-03-27 04:58:51,133 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:58:51,137 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/2151ffec0fe440f2acb7fedb6033c90a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/2151ffec0fe440f2acb7fedb6033c90a 2023-03-27 04:58:51,139 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/ef03f4089e9946a5891f9408aa2ae999 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/ef03f4089e9946a5891f9408aa2ae999 2023-03-27 04:58:51,140 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/283e844b2d1c4cd99de1fd67592124cc to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/283e844b2d1c4cd99de1fd67592124cc 2023-03-27 04:58:51,141 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/7b8b2d649b8a49f6aedc8cd948e1e531 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/7b8b2d649b8a49f6aedc8cd948e1e531 2023-03-27 04:58:51,142 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/c1d2d6cd788c472591d611811df08ef7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/c1d2d6cd788c472591d611811df08ef7 2023-03-27 04:58:51,144 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/fcbe22d7477f490dba0efbe26e76f4d4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/fcbe22d7477f490dba0efbe26e76f4d4 2023-03-27 04:58:51,144 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/86be74830c85442dbda54c9fde901971 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/86be74830c85442dbda54c9fde901971 2023-03-27 04:58:51,145 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/642d532ee37345bb8a64a3102af6af1d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/642d532ee37345bb8a64a3102af6af1d 2023-03-27 04:58:51,146 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/c798d48c7dcf4d55a90620c4cf27ee1e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/c798d48c7dcf4d55a90620c4cf27ee1e 2023-03-27 04:58:51,147 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/3b57dcd96b2b4dbfb3e35c5b42dc7bc3 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/3b57dcd96b2b4dbfb3e35c5b42dc7bc3 2023-03-27 04:58:51,148 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/51fe687a6eae4ddbbaf2168dd462ebb5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/51fe687a6eae4ddbbaf2168dd462ebb5 2023-03-27 04:58:51,149 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/e56de8e34969424bbdb26538d85fdb38 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/e56de8e34969424bbdb26538d85fdb38 2023-03-27 04:58:51,150 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/750129795cd24615922209900dcf1c74 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/750129795cd24615922209900dcf1c74 2023-03-27 04:58:51,151 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/5af325f076ba4cf7a776db0609c314c6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/5af325f076ba4cf7a776db0609c314c6 2023-03-27 04:58:51,152 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/441dd5c3b844405394d811634358c4b0 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/441dd5c3b844405394d811634358c4b0 2023-03-27 04:58:51,153 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/f2e6edd98aba46a1bae0bc55e2b332a4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/f2e6edd98aba46a1bae0bc55e2b332a4 2023-03-27 04:58:51,154 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/bbca2454412a4cc19b82dce057cdf9e3 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/bbca2454412a4cc19b82dce057cdf9e3 2023-03-27 04:58:51,155 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/8c4347c1f03e47b4a4710db33e0cda30 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/8c4347c1f03e47b4a4710db33e0cda30 2023-03-27 04:58:51,156 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/ceb367ec19f54f279db60e6c0033db17 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/ceb367ec19f54f279db60e6c0033db17 2023-03-27 04:58:51,157 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/d70fbc41bdb44fdda644cc6826a2fa12 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/d70fbc41bdb44fdda644cc6826a2fa12 2023-03-27 04:58:51,158 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/66c1cbec50fb4587bdb1af7f9fba2af2 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/66c1cbec50fb4587bdb1af7f9fba2af2 2023-03-27 04:58:51,159 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/71f9243e80fd47948eda363f8e1b726b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/71f9243e80fd47948eda363f8e1b726b 2023-03-27 04:58:51,160 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/7ec88bf754ce429d9e69ed6d989155b9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/7ec88bf754ce429d9e69ed6d989155b9 2023-03-27 04:58:51,161 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/ef4991d24fff4003afea9d49c34a3255 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/ef4991d24fff4003afea9d49c34a3255 2023-03-27 04:58:51,162 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/48b034b9cb98439ea629db2d3483db74 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/48b034b9cb98439ea629db2d3483db74 2023-03-27 04:58:51,165 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/065c94b100dd4400bcabe249a6d62178, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/4bfaed941f4d4c9095e27635ad7c63b7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/a0e314b3af98485e9d5bec29c5e9c459, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/ac96204ceea448c8afbcefe480c607da, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/4e9f632b057e4e3e8f90d5d907f55a4e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/b492350e1347453d94d85a33b0d0d5bd, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/dbd87b621fa7488da717bd351a1dd193, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/7584a31f199846d39ce97150800eca50, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/37a7d8a689cf4e2ea77e1e8168a84588, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/369ffe9420df42cba7a87022c882f6bf, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/d71b50fb7eb74af79fa67aec53c79d3c, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/0df06ca66f8544a0927983c60bb35fb3, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/1cbf8b2767b7414d9cfc13a612c08263, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/b8810a58181b4936bf7a8cf5a71e1d4b, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/020591f377da475289a95065cba92fe9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/792a8834566e4026999d13fd874d99b2, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/0fa5c70f80414a62b86e84231dbf9531, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/f38008770ff6487d949e4c58cb21625f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/518e6513e7d443e1a7bcc154e3a50347, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/f1843cdd8e62467db3a3bc671ff8882a, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/6bf05e3e05554519bde8f5e792fce413, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/38696156fc994a63a139911b7cb11f85, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/ac55100f6b854014bee9fd926957a263, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/eb50cd60f6c344f8a365a8b55439e0eb, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/c0cc737d52994966acbbed39e290af77, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/553d6848e7344b64b2505c4e513bca40] to archive 2023-03-27 04:58:51,166 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:58:51,168 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/065c94b100dd4400bcabe249a6d62178 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/065c94b100dd4400bcabe249a6d62178 2023-03-27 04:58:51,169 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/4bfaed941f4d4c9095e27635ad7c63b7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/4bfaed941f4d4c9095e27635ad7c63b7 2023-03-27 04:58:51,170 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/a0e314b3af98485e9d5bec29c5e9c459 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/a0e314b3af98485e9d5bec29c5e9c459 2023-03-27 04:58:51,170 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/ac96204ceea448c8afbcefe480c607da to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/ac96204ceea448c8afbcefe480c607da 2023-03-27 04:58:51,171 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/4e9f632b057e4e3e8f90d5d907f55a4e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/4e9f632b057e4e3e8f90d5d907f55a4e 2023-03-27 04:58:51,172 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/b492350e1347453d94d85a33b0d0d5bd to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/b492350e1347453d94d85a33b0d0d5bd 2023-03-27 04:58:51,173 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/dbd87b621fa7488da717bd351a1dd193 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/dbd87b621fa7488da717bd351a1dd193 2023-03-27 04:58:51,174 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/7584a31f199846d39ce97150800eca50 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/7584a31f199846d39ce97150800eca50 2023-03-27 04:58:51,174 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/37a7d8a689cf4e2ea77e1e8168a84588 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/37a7d8a689cf4e2ea77e1e8168a84588 2023-03-27 04:58:51,175 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/369ffe9420df42cba7a87022c882f6bf to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/369ffe9420df42cba7a87022c882f6bf 2023-03-27 04:58:51,176 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/d71b50fb7eb74af79fa67aec53c79d3c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/d71b50fb7eb74af79fa67aec53c79d3c 2023-03-27 04:58:51,178 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/0df06ca66f8544a0927983c60bb35fb3 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/0df06ca66f8544a0927983c60bb35fb3 2023-03-27 04:58:51,179 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/1cbf8b2767b7414d9cfc13a612c08263 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/1cbf8b2767b7414d9cfc13a612c08263 2023-03-27 04:58:51,180 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/b8810a58181b4936bf7a8cf5a71e1d4b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/b8810a58181b4936bf7a8cf5a71e1d4b 2023-03-27 04:58:51,181 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/020591f377da475289a95065cba92fe9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/020591f377da475289a95065cba92fe9 2023-03-27 04:58:51,184 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/792a8834566e4026999d13fd874d99b2 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/792a8834566e4026999d13fd874d99b2 2023-03-27 04:58:51,185 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/0fa5c70f80414a62b86e84231dbf9531 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/0fa5c70f80414a62b86e84231dbf9531 2023-03-27 04:58:51,186 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/f38008770ff6487d949e4c58cb21625f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/f38008770ff6487d949e4c58cb21625f 2023-03-27 04:58:51,187 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/518e6513e7d443e1a7bcc154e3a50347 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/518e6513e7d443e1a7bcc154e3a50347 2023-03-27 04:58:51,188 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/f1843cdd8e62467db3a3bc671ff8882a to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/f1843cdd8e62467db3a3bc671ff8882a 2023-03-27 04:58:51,189 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/6bf05e3e05554519bde8f5e792fce413 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/6bf05e3e05554519bde8f5e792fce413 2023-03-27 04:58:51,190 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/38696156fc994a63a139911b7cb11f85 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/38696156fc994a63a139911b7cb11f85 2023-03-27 04:58:51,191 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/ac55100f6b854014bee9fd926957a263 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/ac55100f6b854014bee9fd926957a263 2023-03-27 04:58:51,192 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/eb50cd60f6c344f8a365a8b55439e0eb to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/eb50cd60f6c344f8a365a8b55439e0eb 2023-03-27 04:58:51,193 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/c0cc737d52994966acbbed39e290af77 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/c0cc737d52994966acbbed39e290af77 2023-03-27 04:58:51,194 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/553d6848e7344b64b2505c4e513bca40 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/553d6848e7344b64b2505c4e513bca40 2023-03-27 04:58:51,197 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/c5989fc4514e44e0a7e88a12806eed09, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ce834cc136ea4d2c8bba09c4179bc6a9, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/70addde566d8431b8af5d1725ea5e202, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b9192da1c3a64076ac9a983c7af770cb, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/3cb5dd7214a74be9bc9a79053d6f4bf4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/c04df40d242e461ea799653acf1e7650, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b6ec2edc216f45a4bd81164c2bce3d66, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/fa67c5ca641f47f5a58017c7acbb1c39, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/535e96ea80404e45b9b0f00a8f040ab0, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/20d6c2e543294ba3bd9c0ebe38fcd4a6, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b947a99f60e845e18d80aaf7afa63c58, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/83998e74701a49c99519f06c62636cf5, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/f15cad0a2a424b2ea24de9bc3ac7932e, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/dadea9052e354c4aaafdadd459fe2602, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ad6064b584e04e24b198ed0b87bdce5f, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/a49914e25d7247af8f173e2509499c0d, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/a922eb1d75554ac7a9aeaa6b83e886f4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/5944e6028c8147e38623498945b749a7, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/5764f3f6e6334627ba7b6b74dc063910, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/9f2ecc4b3d434fe0947ebe901f697cae, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ce3aa0157b7b46c49c19d2223e1add58, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/126c63a7929543fc9ffdaeeb71ec6ca1, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/10d248e9144c4c9e9bef5544ff075855, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/1cb93a5dc1fc483a907b06e51a8d98a4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/8e5118ac85164c7ab657e6bb4f46b9e4, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/8d4bbdde4f3b4bc88ff08dc9c02a6900] to archive 2023-03-27 04:58:51,202 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-27 04:58:51,204 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/c5989fc4514e44e0a7e88a12806eed09 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/c5989fc4514e44e0a7e88a12806eed09 2023-03-27 04:58:51,205 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ce834cc136ea4d2c8bba09c4179bc6a9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ce834cc136ea4d2c8bba09c4179bc6a9 2023-03-27 04:58:51,206 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/70addde566d8431b8af5d1725ea5e202 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/70addde566d8431b8af5d1725ea5e202 2023-03-27 04:58:51,207 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b9192da1c3a64076ac9a983c7af770cb to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b9192da1c3a64076ac9a983c7af770cb 2023-03-27 04:58:51,208 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/3cb5dd7214a74be9bc9a79053d6f4bf4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/3cb5dd7214a74be9bc9a79053d6f4bf4 2023-03-27 04:58:51,212 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/c04df40d242e461ea799653acf1e7650 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/c04df40d242e461ea799653acf1e7650 2023-03-27 04:58:51,213 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b6ec2edc216f45a4bd81164c2bce3d66 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b6ec2edc216f45a4bd81164c2bce3d66 2023-03-27 04:58:51,213 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/fa67c5ca641f47f5a58017c7acbb1c39 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/fa67c5ca641f47f5a58017c7acbb1c39 2023-03-27 04:58:51,214 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/535e96ea80404e45b9b0f00a8f040ab0 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/535e96ea80404e45b9b0f00a8f040ab0 2023-03-27 04:58:51,215 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/20d6c2e543294ba3bd9c0ebe38fcd4a6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/20d6c2e543294ba3bd9c0ebe38fcd4a6 2023-03-27 04:58:51,216 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b947a99f60e845e18d80aaf7afa63c58 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/b947a99f60e845e18d80aaf7afa63c58 2023-03-27 04:58:51,217 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/83998e74701a49c99519f06c62636cf5 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/83998e74701a49c99519f06c62636cf5 2023-03-27 04:58:51,218 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/f15cad0a2a424b2ea24de9bc3ac7932e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/f15cad0a2a424b2ea24de9bc3ac7932e 2023-03-27 04:58:51,219 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/dadea9052e354c4aaafdadd459fe2602 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/dadea9052e354c4aaafdadd459fe2602 2023-03-27 04:58:51,220 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ad6064b584e04e24b198ed0b87bdce5f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ad6064b584e04e24b198ed0b87bdce5f 2023-03-27 04:58:51,221 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/a49914e25d7247af8f173e2509499c0d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/a49914e25d7247af8f173e2509499c0d 2023-03-27 04:58:51,222 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/a922eb1d75554ac7a9aeaa6b83e886f4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/a922eb1d75554ac7a9aeaa6b83e886f4 2023-03-27 04:58:51,223 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/5944e6028c8147e38623498945b749a7 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/5944e6028c8147e38623498945b749a7 2023-03-27 04:58:51,224 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/5764f3f6e6334627ba7b6b74dc063910 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/5764f3f6e6334627ba7b6b74dc063910 2023-03-27 04:58:51,225 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/9f2ecc4b3d434fe0947ebe901f697cae to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/9f2ecc4b3d434fe0947ebe901f697cae 2023-03-27 04:58:51,226 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ce3aa0157b7b46c49c19d2223e1add58 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/ce3aa0157b7b46c49c19d2223e1add58 2023-03-27 04:58:51,227 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/126c63a7929543fc9ffdaeeb71ec6ca1 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/126c63a7929543fc9ffdaeeb71ec6ca1 2023-03-27 04:58:51,228 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/10d248e9144c4c9e9bef5544ff075855 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/10d248e9144c4c9e9bef5544ff075855 2023-03-27 04:58:51,228 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/1cb93a5dc1fc483a907b06e51a8d98a4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/1cb93a5dc1fc483a907b06e51a8d98a4 2023-03-27 04:58:51,229 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/8e5118ac85164c7ab657e6bb4f46b9e4 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/8e5118ac85164c7ab657e6bb4f46b9e4 2023-03-27 04:58:51,230 DEBUG [StoreCloser-TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/8d4bbdde4f3b4bc88ff08dc9c02a6900 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/8d4bbdde4f3b4bc88ff08dc9c02a6900 2023-03-27 04:58:51,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/recovered.edits/1512.seqid, newMaxSeqId=1512, maxSeqId=4 2023-03-27 04:58:51,235 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a. 2023-03-27 04:58:51,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for a34144f135b39d6005ad383002fabe3a: 2023-03-27 04:58:51,236 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:51,236 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=a34144f135b39d6005ad383002fabe3a, regionState=CLOSED 2023-03-27 04:58:51,236 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679893131236"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679893131236"}]},"ts":"1679893131236"} 2023-03-27 04:58:51,238 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-27 04:58:51,238 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure a34144f135b39d6005ad383002fabe3a, server=jenkins-hbase20.apache.org,44923,1679892955505 in 1.9700 sec 2023-03-27 04:58:51,239 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-27 04:58:51,239 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a34144f135b39d6005ad383002fabe3a, UNASSIGN in 1.9790 sec 2023-03-27 04:58:51,239 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679893131239"}]},"ts":"1679893131239"} 2023-03-27 04:58:51,240 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-27 04:58:51,241 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-27 04:58:51,242 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9890 sec 2023-03-27 04:58:51,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-27 04:58:51,359 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-27 04:58:51,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-27 04:58:51,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:58:51,361 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:58:51,362 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:58:51,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-27 04:58:51,370 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:51,371 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C, FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/recovered.edits] 2023-03-27 04:58:51,374 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/465c6691746947098b3800a426cf9459 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/465c6691746947098b3800a426cf9459 2023-03-27 04:58:51,375 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/5361bdfd943c4d689379319084db40d1 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/5361bdfd943c4d689379319084db40d1 2023-03-27 04:58:51,376 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/a1e07bb4296e41e689fea876a38a738d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/A/a1e07bb4296e41e689fea876a38a738d 2023-03-27 04:58:51,378 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/98b44bb4e37342d79ece245f9c783b11 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/98b44bb4e37342d79ece245f9c783b11 2023-03-27 04:58:51,379 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/c5e584dcc0e9470d931e7f9b9193e685 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/c5e584dcc0e9470d931e7f9b9193e685 2023-03-27 04:58:51,380 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/d07216f6e728456caabd939599e25e0f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/B/d07216f6e728456caabd939599e25e0f 2023-03-27 04:58:51,382 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/0602f8348e254cecac36ea6e5d8a869f to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/0602f8348e254cecac36ea6e5d8a869f 2023-03-27 04:58:51,383 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/124fe40574fb495abdff5d45733e9fdd to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/124fe40574fb495abdff5d45733e9fdd 2023-03-27 04:58:51,384 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/25ac5d6a0966478995a387adde1baf78 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/C/25ac5d6a0966478995a387adde1baf78 2023-03-27 04:58:51,386 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/recovered.edits/1512.seqid to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a/recovered.edits/1512.seqid 2023-03-27 04:58:51,387 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/.tmp/data/default/TestAcidGuarantees/a34144f135b39d6005ad383002fabe3a 2023-03-27 04:58:51,387 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-27 04:58:51,387 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-27 04:58:51,388 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-27 04:58:51,390 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303270bde693f7628497a9cca732f4bc509f6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303270bde693f7628497a9cca732f4bc509f6 2023-03-27 04:58:51,391 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303270e0e3a1429ad4579a498d59b01f9f8cb to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303270e0e3a1429ad4579a498d59b01f9f8cb 2023-03-27 04:58:51,392 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032726b3714d8d984f9c89faab4929e61a19 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032726b3714d8d984f9c89faab4929e61a19 2023-03-27 04:58:51,393 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032728c203fdb9014483ae9de126174e5bb9 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032728c203fdb9014483ae9de126174e5bb9 2023-03-27 04:58:51,394 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303272eff4ce5b007422d93b5f8b323f1956e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303272eff4ce5b007422d93b5f8b323f1956e 2023-03-27 04:58:51,395 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303274984819bdbad4aca991d567e74464b5b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303274984819bdbad4aca991d567e74464b5b 2023-03-27 04:58:51,396 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303274aecf9e7782641e78cad3e2248552467 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303274aecf9e7782641e78cad3e2248552467 2023-03-27 04:58:51,397 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032753849835d30547d3a323fef1eab93ee3 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032753849835d30547d3a323fef1eab93ee3 2023-03-27 04:58:51,398 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303275aa4c9096f07476abec818270e763e22 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303275aa4c9096f07476abec818270e763e22 2023-03-27 04:58:51,399 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303275f30812cc1444c808499c9613e124a2c to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303275f30812cc1444c808499c9613e124a2c 2023-03-27 04:58:51,400 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303279160337b7d814feeb383ab23e02311f0 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303279160337b7d814feeb383ab23e02311f0 2023-03-27 04:58:51,401 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303279ce9a1c6ff9e412195a238487a46555b to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303279ce9a1c6ff9e412195a238487a46555b 2023-03-27 04:58:51,403 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327a03b2123800d42b6be6c4ebe5bf26c27 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327a03b2123800d42b6be6c4ebe5bf26c27 2023-03-27 04:58:51,404 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327b61b16fb3d8744dfa02208ae982b633e to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327b61b16fb3d8744dfa02208ae982b633e 2023-03-27 04:58:51,405 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327b6946245b2b94664a2d83bf914635800 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327b6946245b2b94664a2d83bf914635800 2023-03-27 04:58:51,407 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327c6f9c97f3783440fa433ad8c0953a2f6 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327c6f9c97f3783440fa433ad8c0953a2f6 2023-03-27 04:58:51,408 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327ca38303d397340b7881b71ffa8194057 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327ca38303d397340b7881b71ffa8194057 2023-03-27 04:58:51,409 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327d0de25cf1c0844fb841837f0e5c42923 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327d0de25cf1c0844fb841837f0e5c42923 2023-03-27 04:58:51,410 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327d19017a1306c4e17969b757b1df8de9d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327d19017a1306c4e17969b757b1df8de9d 2023-03-27 04:58:51,411 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e9d99a1351264a2a8006c4bdafa93c2d to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327e9d99a1351264a2a8006c4bdafa93c2d 2023-03-27 04:58:51,412 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327fac0094cc0ad46eb9eaa0b8f346e7896 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327fac0094cc0ad46eb9eaa0b8f346e7896 2023-03-27 04:58:51,413 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327fc013c80436e4f8b84a16a495a74d370 to hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230327fc013c80436e4f8b84a16a495a74d370 2023-03-27 04:58:51,414 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-27 04:58:51,415 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:58:51,430 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-27 04:58:51,432 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-27 04:58:51,432 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:58:51,433 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-27 04:58:51,433 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679893131433"}]},"ts":"9223372036854775807"} 2023-03-27 04:58:51,434 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-27 04:58:51,434 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a34144f135b39d6005ad383002fabe3a, NAME => 'TestAcidGuarantees,,1679893102581.a34144f135b39d6005ad383002fabe3a.', STARTKEY => '', ENDKEY => ''}] 2023-03-27 04:58:51,434 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-27 04:58:51,434 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679893131434"}]},"ts":"9223372036854775807"} 2023-03-27 04:58:51,435 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-27 04:58:51,438 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-27 04:58:51,439 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 79 msec 2023-03-27 04:58:51,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36293] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-27 04:58:51,463 INFO [Listener at localhost.localdomain/43311] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-27 04:58:51,477 INFO [Listener at localhost.localdomain/43311] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 230), OpenFileDescriptor=565 (was 561) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=459 (was 416) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=704 (was 992) 2023-03-27 04:58:51,478 INFO [Listener at localhost.localdomain/43311] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-27 04:58:51,478 INFO [Listener at localhost.localdomain/43311] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-27 04:58:51,478 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70531455 to 127.0.0.1:61059 2023-03-27 04:58:51,478 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:51,478 DEBUG [Listener at localhost.localdomain/43311] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-27 04:58:51,478 DEBUG [Listener at localhost.localdomain/43311] util.JVMClusterUtil(257): Found active master hash=589204595, stopped=false 2023-03-27 04:58:51,478 INFO [Listener at localhost.localdomain/43311] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,36293,1679892954567 2023-03-27 04:58:51,489 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-27 04:58:51,489 INFO [Listener at localhost.localdomain/43311] procedure2.ProcedureExecutor(629): Stopping 2023-03-27 04:58:51,489 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-27 04:58:51,490 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-27 04:58:51,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-27 04:58:51,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-27 04:58:51,490 DEBUG [Listener at localhost.localdomain/43311] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bb49698 to 127.0.0.1:61059 2023-03-27 04:58:51,490 DEBUG [Listener at localhost.localdomain/43311] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:51,490 INFO [Listener at localhost.localdomain/43311] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,44923,1679892955505' ***** 2023-03-27 04:58:51,490 INFO [Listener at localhost.localdomain/43311] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-27 04:58:51,491 INFO [RS:0;jenkins-hbase20:44923] regionserver.HeapMemoryManager(220): Stopping 2023-03-27 04:58:51,491 INFO [RS:0;jenkins-hbase20:44923] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-27 04:58:51,491 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-27 04:58:51,491 INFO [RS:0;jenkins-hbase20:44923] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-27 04:58:51,491 INFO [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(3304): Received CLOSE for 44fb63855dd55c4d7b8c984f4960cd12 2023-03-27 04:58:51,492 INFO [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:51,492 DEBUG [RS:0;jenkins-hbase20:44923] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x140e3e31 to 127.0.0.1:61059 2023-03-27 04:58:51,493 DEBUG [RS:0;jenkins-hbase20:44923] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:51,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 44fb63855dd55c4d7b8c984f4960cd12, disabling compactions & flushes 2023-03-27 04:58:51,493 INFO [RS:0;jenkins-hbase20:44923] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-27 04:58:51,493 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12. 2023-03-27 04:58:51,493 INFO [RS:0;jenkins-hbase20:44923] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-27 04:58:51,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12. 2023-03-27 04:58:51,493 INFO [RS:0;jenkins-hbase20:44923] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-27 04:58:51,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12. after waiting 0 ms 2023-03-27 04:58:51,493 INFO [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-27 04:58:51,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12. 2023-03-27 04:58:51,493 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 44fb63855dd55c4d7b8c984f4960cd12 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-27 04:58:51,494 INFO [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-27 04:58:51,494 DEBUG [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(1479): Online Regions={44fb63855dd55c4d7b8c984f4960cd12=hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12., 1588230740=hbase:meta,,1.1588230740} 2023-03-27 04:58:51,494 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-27 04:58:51,494 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-27 04:58:51,495 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-27 04:58:51,495 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-27 04:58:51,495 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-27 04:58:51,495 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-03-27 04:58:51,496 DEBUG [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(1505): Waiting on 1588230740, 44fb63855dd55c4d7b8c984f4960cd12 2023-03-27 04:58:51,502 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/.tmp/info/0d230f1ff198429ca4a8c6a0a011a8aa 2023-03-27 04:58:51,506 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0d230f1ff198429ca4a8c6a0a011a8aa 2023-03-27 04:58:51,525 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-27 04:58:51,696 DEBUG [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(1505): Waiting on 1588230740, 44fb63855dd55c4d7b8c984f4960cd12 2023-03-27 04:58:51,897 DEBUG [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(1505): Waiting on 1588230740, 44fb63855dd55c4d7b8c984f4960cd12 2023-03-27 04:58:51,902 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/namespace/44fb63855dd55c4d7b8c984f4960cd12/.tmp/info/c598a9e02b4648b692c2441d1daf5273 2023-03-27 04:58:51,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/namespace/44fb63855dd55c4d7b8c984f4960cd12/.tmp/info/c598a9e02b4648b692c2441d1daf5273 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/namespace/44fb63855dd55c4d7b8c984f4960cd12/info/c598a9e02b4648b692c2441d1daf5273 2023-03-27 04:58:51,912 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/.tmp/rep_barrier/18e0bf56e2bf4e6e88da7b00635977e0 2023-03-27 04:58:51,914 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/namespace/44fb63855dd55c4d7b8c984f4960cd12/info/c598a9e02b4648b692c2441d1daf5273, entries=2, sequenceid=6, filesize=4.8 K 2023-03-27 04:58:51,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 44fb63855dd55c4d7b8c984f4960cd12 in 422ms, sequenceid=6, compaction requested=false 2023-03-27 04:58:51,916 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 18e0bf56e2bf4e6e88da7b00635977e0 2023-03-27 04:58:51,923 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/namespace/44fb63855dd55c4d7b8c984f4960cd12/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-27 04:58:51,927 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12. 2023-03-27 04:58:51,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 44fb63855dd55c4d7b8c984f4960cd12: 2023-03-27 04:58:51,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1679892957651.44fb63855dd55c4d7b8c984f4960cd12. 2023-03-27 04:58:52,097 DEBUG [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-27 04:58:52,112 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-03-27 04:58:52,112 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-03-27 04:58:52,297 DEBUG [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-27 04:58:52,328 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/.tmp/table/65c402ae30ae413bbc284d9672cb8ccb 2023-03-27 04:58:52,331 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 65c402ae30ae413bbc284d9672cb8ccb 2023-03-27 04:58:52,331 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/.tmp/info/0d230f1ff198429ca4a8c6a0a011a8aa as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/info/0d230f1ff198429ca4a8c6a0a011a8aa 2023-03-27 04:58:52,334 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0d230f1ff198429ca4a8c6a0a011a8aa 2023-03-27 04:58:52,334 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/info/0d230f1ff198429ca4a8c6a0a011a8aa, entries=22, sequenceid=93, filesize=7.1 K 2023-03-27 04:58:52,335 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/.tmp/rep_barrier/18e0bf56e2bf4e6e88da7b00635977e0 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/rep_barrier/18e0bf56e2bf4e6e88da7b00635977e0 2023-03-27 04:58:52,338 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 18e0bf56e2bf4e6e88da7b00635977e0 2023-03-27 04:58:52,338 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/rep_barrier/18e0bf56e2bf4e6e88da7b00635977e0, entries=6, sequenceid=93, filesize=5.5 K 2023-03-27 04:58:52,339 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/.tmp/table/65c402ae30ae413bbc284d9672cb8ccb as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/table/65c402ae30ae413bbc284d9672cb8ccb 2023-03-27 04:58:52,341 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 65c402ae30ae413bbc284d9672cb8ccb 2023-03-27 04:58:52,341 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/table/65c402ae30ae413bbc284d9672cb8ccb, entries=9, sequenceid=93, filesize=5.5 K 2023-03-27 04:58:52,342 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 847ms, sequenceid=93, compaction requested=false 2023-03-27 04:58:52,354 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-27 04:58:52,355 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-27 04:58:52,356 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-27 04:58:52,356 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-27 04:58:52,356 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-27 04:58:52,497 INFO [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,44923,1679892955505; all regions closed. 2023-03-27 04:58:52,507 DEBUG [RS:0;jenkins-hbase20:44923] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/oldWALs 2023-03-27 04:58:52,507 INFO [RS:0;jenkins-hbase20:44923] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C44923%2C1679892955505.meta:.meta(num 1679892957395) 2023-03-27 04:58:52,518 DEBUG [RS:0;jenkins-hbase20:44923] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/oldWALs 2023-03-27 04:58:52,518 INFO [RS:0;jenkins-hbase20:44923] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C44923%2C1679892955505:(num 1679892957255) 2023-03-27 04:58:52,518 DEBUG [RS:0;jenkins-hbase20:44923] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:52,518 INFO [RS:0;jenkins-hbase20:44923] regionserver.LeaseManager(133): Closed leases 2023-03-27 04:58:52,518 INFO [RS:0;jenkins-hbase20:44923] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-03-27 04:58:52,519 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-27 04:58:52,522 INFO [RS:0;jenkins-hbase20:44923] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:44923 2023-03-27 04:58:52,528 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-27 04:58:52,528 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,44923,1679892955505 2023-03-27 04:58:52,528 ERROR [Listener at localhost.localdomain/43311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@166ea729 rejected from java.util.concurrent.ThreadPoolExecutor@1251213c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2672] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-27 04:58:52,529 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-27 04:58:52,529 ERROR [Listener at localhost.localdomain/43311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@57a769e rejected from java.util.concurrent.ThreadPoolExecutor@1251213c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2672] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-27 04:58:52,530 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,44923,1679892955505] 2023-03-27 04:58:52,531 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,44923,1679892955505; numProcessing=1 2023-03-27 04:58:52,532 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,44923,1679892955505 already deleted, retry=false 2023-03-27 04:58:52,532 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,44923,1679892955505 expired; onlineServers=0 2023-03-27 04:58:52,532 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,36293,1679892954567' ***** 2023-03-27 04:58:52,532 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-27 04:58:52,532 DEBUG [M:0;jenkins-hbase20:36293] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62238648, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-03-27 04:58:52,532 INFO [M:0;jenkins-hbase20:36293] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,36293,1679892954567 2023-03-27 04:58:52,532 INFO [M:0;jenkins-hbase20:36293] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,36293,1679892954567; all regions closed. 2023-03-27 04:58:52,533 DEBUG [M:0;jenkins-hbase20:36293] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-27 04:58:52,533 DEBUG [M:0;jenkins-hbase20:36293] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-27 04:58:52,533 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-27 04:58:52,533 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1679892956918] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1679892956918,5,FailOnTimeoutGroup] 2023-03-27 04:58:52,533 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1679892956917] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1679892956917,5,FailOnTimeoutGroup] 2023-03-27 04:58:52,533 DEBUG [M:0;jenkins-hbase20:36293] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-27 04:58:52,533 INFO [M:0;jenkins-hbase20:36293] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-27 04:58:52,533 INFO [M:0;jenkins-hbase20:36293] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-27 04:58:52,533 INFO [M:0;jenkins-hbase20:36293] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-03-27 04:58:52,534 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-27 04:58:52,534 DEBUG [M:0;jenkins-hbase20:36293] master.HMaster(1512): Stopping service threads 2023-03-27 04:58:52,534 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-27 04:58:52,534 INFO [M:0;jenkins-hbase20:36293] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-27 04:58:52,535 ERROR [M:0;jenkins-hbase20:36293] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (485495027) connection to localhost.localdomain/127.0.0.1:41275 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-27 04:58:52,535 INFO [M:0;jenkins-hbase20:36293] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-27 04:58:52,535 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-27 04:58:52,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-27 04:58:52,536 DEBUG [M:0;jenkins-hbase20:36293] zookeeper.ZKUtil(398): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-27 04:58:52,536 WARN [M:0;jenkins-hbase20:36293] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-27 04:58:52,536 INFO [M:0;jenkins-hbase20:36293] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-27 04:58:52,536 INFO [M:0;jenkins-hbase20:36293] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-27 04:58:52,537 DEBUG [M:0;jenkins-hbase20:36293] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-27 04:58:52,537 INFO [M:0;jenkins-hbase20:36293] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-27 04:58:52,537 DEBUG [M:0;jenkins-hbase20:36293] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-27 04:58:52,537 DEBUG [M:0;jenkins-hbase20:36293] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-27 04:58:52,537 DEBUG [M:0;jenkins-hbase20:36293] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-27 04:58:52,537 INFO [M:0;jenkins-hbase20:36293] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.37 KB heapSize=393.97 KB 2023-03-27 04:58:52,562 INFO [M:0;jenkins-hbase20:36293] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.37 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/a919fea876b2442189dede2c48427a30 2023-03-27 04:58:52,572 INFO [M:0;jenkins-hbase20:36293] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a919fea876b2442189dede2c48427a30 2023-03-27 04:58:52,573 DEBUG [M:0;jenkins-hbase20:36293] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/a919fea876b2442189dede2c48427a30 as hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/a919fea876b2442189dede2c48427a30 2023-03-27 04:58:52,576 INFO [M:0;jenkins-hbase20:36293] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a919fea876b2442189dede2c48427a30 2023-03-27 04:58:52,577 INFO [M:0;jenkins-hbase20:36293] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41275/user/jenkins/test-data/6ddaca76-6b87-bfba-e5d7-0a7a916bcc8c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/a919fea876b2442189dede2c48427a30, entries=65, sequenceid=578, filesize=24.7 K 2023-03-27 04:58:52,577 INFO [M:0;jenkins-hbase20:36293] regionserver.HRegion(2948): Finished flush of dataSize ~345.37 KB/353658, heapSize ~393.95 KB/403408, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 40ms, sequenceid=578, compaction requested=false 2023-03-27 04:58:52,579 INFO [M:0;jenkins-hbase20:36293] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-27 04:58:52,579 DEBUG [M:0;jenkins-hbase20:36293] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-27 04:58:52,585 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-27 04:58:52,585 INFO [M:0;jenkins-hbase20:36293] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-27 04:58:52,585 INFO [M:0;jenkins-hbase20:36293] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:36293 2023-03-27 04:58:52,586 DEBUG [M:0;jenkins-hbase20:36293] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,36293,1679892954567 already deleted, retry=false 2023-03-27 04:58:52,631 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-27 04:58:52,631 INFO [RS:0;jenkins-hbase20:44923] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,44923,1679892955505; zookeeper connection closed. 2023-03-27 04:58:52,631 ERROR [Listener at localhost.localdomain/43311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7d8d5bc3 rejected from java.util.concurrent.ThreadPoolExecutor@1251213c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2672] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-27 04:58:52,632 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@66c02b80] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@66c02b80 2023-03-27 04:58:52,632 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): regionserver:44923-0x100512252fd0001, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-27 04:58:52,632 ERROR [Listener at localhost.localdomain/43311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@61e40caa rejected from java.util.concurrent.ThreadPoolExecutor@1251213c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2672] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-27 04:58:52,632 INFO [Listener at localhost.localdomain/43311] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-27 04:58:52,731 INFO [M:0;jenkins-hbase20:36293] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,36293,1679892954567; zookeeper connection closed. 2023-03-27 04:58:52,731 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-27 04:58:52,731 ERROR [Listener at localhost.localdomain/43311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1e85f2db rejected from java.util.concurrent.ThreadPoolExecutor@6255fcf7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4467] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-27 04:58:52,732 DEBUG [Listener at localhost.localdomain/43311-EventThread] zookeeper.ZKWatcher(600): master:36293-0x100512252fd0000, quorum=127.0.0.1:61059, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-27 04:58:52,732 ERROR [Listener at localhost.localdomain/43311-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@58c5fd7 rejected from java.util.concurrent.ThreadPoolExecutor@6255fcf7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4467] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-27 04:58:52,733 WARN [Listener at localhost.localdomain/43311] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-27 04:58:52,736 INFO [Listener at localhost.localdomain/43311] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-27 04:58:52,847 WARN [BP-1207162225-148.251.75.209-1679892951960 heartbeating to localhost.localdomain/127.0.0.1:41275] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-27 04:58:52,847 WARN [BP-1207162225-148.251.75.209-1679892951960 heartbeating to localhost.localdomain/127.0.0.1:41275] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1207162225-148.251.75.209-1679892951960 (Datanode Uuid 093b1e1f-e2c0-4364-97ab-7016262b240a) service to localhost.localdomain/127.0.0.1:41275 2023-03-27 04:58:52,850 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/cluster_6fb3ee64-1fe6-4ac3-e6f2-f169c384bd1a/dfs/data/data1/current/BP-1207162225-148.251.75.209-1679892951960] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-27 04:58:52,853 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fe7117e3-fadd-9638-07af-6169b20534df/cluster_6fb3ee64-1fe6-4ac3-e6f2-f169c384bd1a/dfs/data/data2/current/BP-1207162225-148.251.75.209-1679892951960] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-27 04:58:52,908 INFO [Listener at localhost.localdomain/43311] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-03-27 04:58:53,071 INFO [Listener at localhost.localdomain/43311] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-27 04:58:53,266 INFO [Listener at localhost.localdomain/43311] hbase.HBaseTestingUtility(1293): Minicluster is down